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ABSTRACT:  This  dissertation  reports  the  development  of  digital  computer 
techniques  for  detecting  changes  in  scenes  by  normalizing  and 
comparing  pictures  which  were  taken  from  different  camera 
positions  and  under  different  conditions  of  illumination.  The 
pictures  are  first  geometrically  normalized  to  a  common  point 
of  view.  Then  they  are  photometrically  normalized  to  eliminate 
the  differences  due  to  different  illumination,  camera  character¬ 
istics,  and  reflectance  properties  of  the  scene  due  to  different 
sun  and  view  angles.  These  pictures  are  then  geometrically 
registered  by  maximizing  the  cross  correlation  between  areas  in 
them.  The  final  normalized  and  registered  pictures  are  then 
differenced  point  by  point. 

The  geometric  normalization  techniques  require  relatively 
accurate  geometric  models  for  the  camera  and  the  scene,  and 
static  spatial  features  must  be  present  in  the  pictures  to  allow 
precise  geometric  alignment  using  the  technique  of  cross  correla¬ 
tion  maximation. 

Photometric  normalization  also  requires  a  relatively  accurate 
model  for  the  photometric  response  of  the  camera,  a  reflective 
model  for  the  scene  '  reflectance  as  a  function  of  the  illumina¬ 
tion  view,  and  phase  angles'  and  some  assumptions  about  the 
kinds  of  reflectance  changes  which  are  to  be  detected. 
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These  techniques  have  been  incorporated  in  a  system  for 
comparing  Mariner  1971  pictures  of  Mars  to  detect  variable 
surface  phenomena  as  well  as  color  and  polarization  differ¬ 
ences  .  The  system  has  been  tested  using  Mariner  6  and  7 
pictures  of  Mars . 

Although  the  techniques  described  in  this  dissertation  were 
developed  for  Mars  pictures,  their  use  is  not  limited  to 
this  application.  Various  parts  of  this  software  package, 
which  was  developed  for  interactive  use  on  the  time-sharing 
system  of  the  Stanford  Artificial  Intelligence  Laboratory, 
are  currently  being  applied  to  other  scenery. 
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CHAPTER  I 


INTRODUCTION 

Th*  general  picture  comparison  problem  Is  to 
geometrical!^  and  photometrically  normalize  and  register 
Images  so  that  true  c,  1  f  f  *?renoes  |n  the  scene  can  be 
determined.  rather  than  differences  In  conditions  of 
viewing,  and  to  analyze  these  differences, 

The  geometric  analysis  of  photographs  belongs  to  the 
science  of  analytical  photogramrnetry  (Doyle  C1966J),  which 
generally  deals  with  the  t r  I  angu  I  a 1 1 on  of  aerial  photographs 
to  generate  topographic  maos,  Most  results  In  this  f|e|d, 
however,  are  approximations  which  solve  suitably  linearized 
versions  of  the  equations,  Even  neglecting  this  deficiency, 
these  techniques  are  of  little  use  except  when  extremely 
accurate  models  exist  for  the  camera. 

Military  aer|a|  reconna I sance  probably  represents  the 
major  current  application  of  Picture  comparison  technology, 
Most  of  the  known  techniques  |n  this  field  re|y  on  manually 
operated,  analog  Image  correlators,  and  various  other  analog 


hardware.  There  Is  | 1 1 1 1  •  In  the  I  1 1«  ratur •  to  describe 
whet  digital  computer  techniques,  If  any,  have  been 
developed  for  this  purpose. 

The  system  described  In  this  dissertation  le  the  only 
system  known  to  solve  the  problems  of  image  normalization 
and  registration, 

I , A  THE  MARINER  197i  MARS  MISSION  Ci 3 

The  primary  objectlvee  of  the  Mariner  Mare  1971  Project 
are  the  observation  and  mapping  of  Mars  by  two  orbiting 
spacecraft  beginning  |n  November  of  1971  and  continuing  for 
at  least  90  days  (JPL  C 1 9  7  0  3  ) , 

An  orblter  has  the  advantage  over  flyby  missions,  such 


C 1 3  Mariner  8  was  launohed  from  Cape  Kennedy  on  May  8,  1971, 
and  failed  to  make  Earth  orbit,  Therefore,  the  nominal 
Mariner  '71  mission  plans  must  be  ohanged,  It  Is  not  known 
at  this  time  whether  the  Variable  Features  Mission,  which  's 
the  purpose  of  th|s  research,  will  be  attempted,  Since  most 
of  this  dissertation  was  written  bsfore  May  8,  many  of  the 
verb  tenses  referring  to  the  Mariner  '71  Mission  should  be 
ohanged  to  refleot  the  current  situation, 


2 


•g hhibwe 


as  the  Mariner  Mars  1964  and  1969  missions*  that 
observations  from  orbit  oan  ba  mada  ovar  an  extended  period 
of  time#  thus  permitting  the  study  of  temporal  changes  on 
the  Martian  surface, 


The  Mariner  IV,  VI#  and  VIJ  missions  could  make  only 
limited  observations  of  the  surface  of  Mare  because  they 
were  In  the  planet's  vicinity  for  a  very  short  time,  To 
accumulate  more  extensive  data  It  Is  necessary  to  orbit  Mars 
as  planned  In  the  Mariner  ' 71  Mission,  or  orb|t  and  land  as 
planned  In  the  1975  V|k|ng  mission, 


Th*  two  Identical  Mariner  '71  spacecraft  w 


perform 


separata  missions  designated  as  missions  A  and  B,  Mission  A 
Is  primarily  devoted  to  routine  mapping,  attempting  to  view 
a  large  portion  of  the  surface  of  Mare  w|th  the  highest 
ooss I b 1 e  resolution, 


Mission  B  |s  primarily  devoted  to  studying  time 
variable  features  of  the  Martian  atmosphere  and  surface,  It 
will  utilize  an  orb|t  which  w | I |  give  repeated  coverage  of 
several  different  surface  areas  under  essentially  constant 
Illumination  angle*  view  angle,  and  spacecraft  altitude 
(Sagan  C1969]),  with  such  an  orbit,  |t  Is  possible  to  study 
these  areas  of  the  surface  to  detect  changes  during  the 
90-oay  mission, 


The  types  of  changes  which  are  expsoted  art  either 
transient  or  dlurnally  or  seasonally  recurrent,  Translant 
phenomena  that  have  baan  obsarvad  on  Mars  Inoluda  clouds* 
hazes*  and  bright  spots,  Whit*  clouds  have  baan  laan  of  a  I  I 
slzas  and  shapes*  from  terminator  haza  lasting  a  faw  hours 
to  dense*  1200-ml  |a  giants  lasting  days  or  waaks,  Yellow 
clouds  vary  from  small*  dansa*  oranga  or  yellow  objects 
lasting  from  one  to  a  faw  days*  to  objects  which  start  large 
and  grow  larger  until  they  become  a  yellow  vail  covering 
most  of  Mars  and  lasting  a  month  or  more,  Thssa  yellow 
clouds  are  almost  universally  acoepted  as  being  dust  olouds, 
The  Mariner  '71  television  observations  will  yield  data  on 
atmospheric  o|rcu|atlon  by  following  cloud  movement, 

The  most  obvious  seasonal  changes  on  the  Martian 
eurface  Involve  the  polar  caps  and  the  wave  of  darkening, 
The  surface  also  exhibits  seasonal  changes  In  color  and  even 
changes  |n  the  size*  shape*  and  Internal  appearance  of  the 
various  dark  areas  on  the  planet* 

The  polar  eaps  are  believed  to  be  deposits  of  solid 
oarbon  dioxide  condensing  during  the  fall  and  winter  In  each 
hemisphere  and  then  subliming  during  the  spring  and  summer, 
From  many  biological  Points  of  view*  the  receding  polar  cap 
Is  a  looale  of  great  Interest*  and  will  be  observed  during 
the  Mariner  '71  mission, 
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The  wave  of  darkening  Is  Probably  Mars'  graatest  enigma 
and  Is  tha  most  dynamic  event  on  the  o|anet,  It  has  bean 
daacrlbad  as  a  progressiva  dacllna  |n  tha  raflaetlvlty  of 
tha  dark  surface  areas  (and  Inoraasa  In  contrast  with 
aurrounolng  bright  areas)  starting  |n  local  springtime  from 
the  edge  of  the  vaporizing  polar  eaoi  and  moving  toward  and 
across  the  equator.  Whether  tha  darkening  actually  occurs 
as  a  "wave"  from  the  po|a  has  been  argued,  This  darkening 
might  also  be  accompanied  by  eo|or  and  polarization  changes, 
According  to  the  biological  explanation  for  these  seasonal 
changes,  Martian  organisms  Inhabit  the  dark  regions,  and 
their  springtime  growth  In  response  to  tha  Increased 
temperatures  and  humidities  is  the  oause  of  the  darkening 
events,  Several  alternative  non-b  |  o  |  og I ca I  hypotheses  have 
been  proposed.  Including  one  |n  whloh  seasonal  changes  |n 
wlno  Patterns  (due  either  to  meridional  circulation  or 
dust-devils)  redistribute  the  particle  sizes  In  the  bright 
and  dark  areas  and  produce  the  albedo  changes, 

In  selecting  s c I e n 1 1 f I o  objectives  for  the  Mariner  '71 
Mission,  the  study  of  the  wave  of  darkening  was  singled  out 
for  special  cons  I  derat  I  on,  since  during  the  1970  to  1990 
decade,  the  wave  of  darkening  In  the  southern  hemisphere  can 
only  be  observed  |n  1971,  The  southern  hemisphere  contains 
most  of  the  permanently  dark  regions  and  Is  considered  the 
more  Interesting  hemisphere  In  which  to  study  this 


phenomenon, 


Th»  two  spacecraft  irt  Identical  and  contain  a  variety 
of  scientific  Instruments  Including  two  television  cameras, 
an  ultraviolet  spectrometer ,  an  Infrared  Interferometer,  and 
an  Infrared  radiometer,  In  terms  of  volume  of  data 
collected,  the  television  experiment  data  w| I |  outweigh  all 
other  data  by  several  orders  of  magnitude, 

The  two  television  cameras  are  designated  as  cameras  A 
and  b,  The  A  camera  has  a  fooal  length  of  50  mm,  giving  It 
an  11  by  14  degree  angular  field  of  v|ew,  The  B  camera  has 
a  fooal  length  of  500  mm  giving  It  a  1,1  by  1,4  degree  field 
of  view  and  henoe  10  times  the  resolution,  Camera  A  has  a 
oolleotion  of  8  oc|or  and  polarization  filters  whloh  oan  be 
arbitrarily  seleoted,  The  B  oamera  has  a  single  fixed 
(minus  blue)  color  filter,  The  wave  of  darkening  and  other 
variable  surfaoe  features  w|||  be  studied  primarily  using  A 
oamera  olotures  which  are  taken  before  perlapsle  when  the 
sun  and  spaoeoraft  vectors  are  nearly  vertloal  to  the 
surface  |n  the  area  being  photographed,  thus  minimizing 
uncertainties  due  to  light  soatterlng,  host  of  these 
olotures  w| | |  be  taken  under  an  orange  color  filter  which, 
on  the  basle  of  ground  obeervat | ons,  should  make  It  easiest 
to  detect  ohanges  In  albedo, 

The  television  oamerae  are  vldloons,  which  are 
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shuttered  at  speeds  ranging  from  6  to  192  m I  I | | seconds ,  The 
Imaga  on  tha  vldlcon  target  Is  digitized  In  saconds  |n  a 
format  of  9  bits  oar  point*  832  points  per  line,  and  75*0 
lines  per  frame,  0|g!t|zed  samples  are  stored  on  a  digital 
magnetic  tape  recorder  which  oan  nold  approximately  36 
Pictures  until  It  1$  possible  to  transmit  the  pictures  to 
Earth, 

Each  spacecraft  has  a  radio  receiver  whloh  Is  used  to 
oontrol  suoh  things  as  propulsion*  scan  platform  slewing, 
color  filter  selection,  camera  shuttering,  and  tape  recorder 
playback  (and  transmission),  Each  apaceoraft  also  contains 
a  10  or  20  watt  2300  MH*  radio  transmitter  which  during  most 
of  the  mission  w|||  be  able  to  transmit  experiment  and 
spaceoraft  data  to  the  210  foot  Goldstone  antenna  at  a  16 
kbps  (kllo-blts  per  seoond)  rate,  At  this  rate,  each 
oloture  requires  about  5  minutes  transmission  time,  A  dally 
tape  recorder  load  consists  of  about  36  pictures,  whloh  will 
reaulre  a  transmission  time  of  3  hours,  This  is  about  the 
longest  period  of  continuous  communication  between  the 
spacecraft  and  Goldstone  whloh  can  b«  guaranteed  on  every 
orbit,  (Br | ggs  C 1971 3 ) , 

To  maximize  the  scientific  return  from  the  Mariner  '71 
mission,  It  Is  necessary  to  control  the  picture  taking 
sequence  to  concentrate  surfac®  ooverage  In  those  surface 
areas  which  show  the  greatest  variability,  Th|s  requires 
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that  ploturas  received  at  Earth  be  analyzed  »9  Quickly  ario 
tans  !  1 1  va  |  y  aa  possible  to  datact  differences  from  previous 
plotures  of  tha  area,  This  dissertation  Is  tha  result 
of  rasaaroh  In  tha  development  of  digital  computer 
techniques  to  perform  auoh  analysis  on  pictures  from  tha 
Mariner  '71  mission, 

I,B  A  SOLUTION  TO  THE  MARS  PICTURE  COMPARISON  PROBLEM 

In  response  to  tha  needs  of  tha  Mariner  ’IX  Variable 
Features  Team,  Image  processing  *echn|ques  have  been 
developed  by  the  author  at  the  Stanford  University 
A  r  1 1  f  I  o  I  a  |  Intelligence  Project,  utilizing  a.  POP-10 
Interactive  time-sharing  oomoutef  system,  These  techniques 
have  been  Integrated  Into  a  system  which  compares  pairs  of 
Images  taken  at  different  times,  from  different  spacecraft 
positions,  and  perhaps  even  different  spacecraft, 

Figure  1-1  shows  the  structure  of  the  system,  In  this 
figure,  the  rectangular  blocks  Indicate  programs  which 
Implement  the  teohnlaues  deioflbed  In  this  paper,  Where 
appropriate,  the  blook  contains  the  chapter  or  seotlon 
number  where  the  technique  Is  described.  Oval  blooks 
Indicate  data  <usu*lly  ploturas), 

GEOMETRIC  NORMALIZATION)  Since  the  Images  are  different 
...■Beet  I ve  views  of  a  spheroid#  with  no  surface  elevations 


fiTtni±w****--**‘*~'*r"l*‘i»* 
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assumed,  t  geometric  norm*  I  I  zat  |  on  la  required  to  r>i|»tj 
oommon  surfaoe  Points.  In  partloular,  t*ch  Image  Is 
transformed  to  the  same  orthoflraph|c  projection  of  th* 
spherolo.  If  there  wars  no  errors  In  our  knowledge  of  the 
spacecraft  position  and  or|entat|on»  and  no  geometric 
distortions  In  the  optical  and  e|eotronlo  systems  of  the  TV 
camera,  and  If  our  object  were  a  perfect  spheroid#  then  the 
two  normalized  (mages  should  be  In  exact  geometric 
cor r esoondenoe ,  Chapter  II  describes  geometric 
norma  I  I zat I  on  In  detail. 


GEOMETRIC  REGISTRATION!  Unfortunately,  many  sources  of 
geometric  errors  exist.  with  spacecraft  orientation 
contributing  the  largest  error,  In  order  to  remove  these 
geometric  alignment  errors,  It  Is  necessary  to  align 
geometrically  cor r esoond I  no  areas  (features)  In  the  two 
Images.  A  technique  was  developed  which  displaces  one  Image 

I  relative  to  the  other,  searohlng  for  a  (dx,  dy)  translation 

vector  whloh  maximizes  the  cross  correlation  of  the  Images 
over  a  specified  area,  Maximum  cross  correlation  ocours 
when  the  Images  are  properly  rtglstsred  In  that  area. 

I 

If  we  know  the  <dx,  dy)  translation  necessary  to 
properly  register  every  Point  In  the  geometr I oa I | y 
normalized  Images,  then  we  oan  analyze  the  Images  point  by 
point  for  differences, 
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The  assumption  Is  made  that  the  m I sreg I strat I  on  vector 
as  a  sgfunetlon  of  position  In  tne  Image*  Is  a  smooth 
continuous  functlonf  which  can  bs  mods|ad  by  low  ordsr 
polynomials  |n  two  variables,  This  assumption  Is  rsallstlc 
If  the  irode  |  of  the  object  Is  accurate  U.ec  surface 
elevations  are  small  or  view  ana|es  are  similar)  and  If 
optical  and  electronic  distortions  are  smooth  and 
oont I nuous • 

Jn  practice,  when  we  perform  a  least  squares  f|t  of  |ow 
order  polynomials  In  two  variables  to  a  set  of  optimal 
translation  vectors  on  normalized  Mariner  6  and  7  Images,  we 
get  very  small  residual  errors,  In  particular,  when  fitting 
1st  order  polynomials  (whloh  have  3  degrees  of  freedom)  to 
from  10  to  100  data  points,  we  usually  get  residual 
alignment  errors  of  less  than  one  cloture  unit  (Pixel) 
standard  deviation,  This,  at  least  empirically,  Indloates 
that  the  predominant  souroes  of  error  cause  a  smooth  and 
continuous  misalignment  between  the  two  Images,  For  Mariner 
6  and  7,  the  known  geometric  errors  are  primarily  In 
spacecraft  (camera)  orientation,  which  would  oause  primarily 
a  translational,  and  to  a  lesser  degree,  rotational 
misalignment  between  two  Images,  and  therefore  1st  order 
polynomials  |n  two  variables  are  a  good  approximation, 

Having  an  accurate  model  for  the  mlsal Ignment  between 
the  two  Images,  we  can  recalculate  the  orthographic 
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projection  for  on*  0f  tht  Images,  taking  into  account  f,he 
m 1 3a  I  I  gnment  model,  Th#  resulting  Images  will  be  registered 
to  the  accuracy  of  the  m 1 sreg  I  atrat | on  model,  Chapter  m 
describes  geometrlo  registration  In  detail. 

PHOTOMETRIC  NORMALIZATION!  S|no*  the  Images  are  taken 
under  different  Illumination  and  v|e*  anp<es,  a  photometric 
correction  Is  reaulrsd  to  relate  light  Intensity  levels 
reoelved  at  the  camera  to  albedo  on  the  surfaoe,  If  tnere 
were  no  errors  In  our  knowledge  of  the  light  scattering 
function  at  each  location  on  the  p|an*t,  and  no  errors  In 
the  Photometric  resDonae  of  the  vldlcon,  then,  In  theory,  w« 
should  be  able  tp  oreolsefy  determine  the  albedo  at  each 
point  In  th*  Images  and  perform  albedo  comparisons  to  detect 
variable  features,  Chapter  Iv  describes  photometric 
norma  I  1 zat Ion  In  detail, 

PHOTOMETRIC  REGISTRATION!  Detecting  variable  features 
necessitates  a  high  degree  of  photometrlo  acouraoy,  since 
some  of  the  variations  antlolpated  are  relatively  small  (5% 
or  less)  albedo  changes  over  rather  large  areas  of  the 
planet,  These  albedo  changes  may  Increase  the  contrast 
between  two  arsas,  or  may  only  change  the  average  absolute 
light  level  In  both  areas,  If  the  errors  and/or  no|s*  |n 
th*  calibration  of  the  camera  eystem  are  larger  than  the 
albedo  ohanges  to  be  deteotedi  then  we  must  Improve  the 
calibration  using  Information  In  th*  Images, 
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The  known  repeatable  souroes  of  photometric  errors  are 
such  thlnas  as  vldjoon  shading  (non-uniform  rssponss)  and 
residual  Images*  both  of  wh|oh  oan  be  handled  In  a 
systematic  manner  C13(  The  primary  sources  of  error  whloh 
cannot  be  predloted  ar®  an  absolute  gain  error*  and  an 
absolute  offset  error  resembling  soattered  light,  These 
trrors*  whloh  may  be  caused  ®|ttjer  by  errors  In  camera 
calibration*  or  by  errors  In  the  light  scattering  model  for 
the  surface*  give  a  system  output  Y  as  a  function  of  light 
Input  X  as»  Y  *  aX  +  b, 

Using  this  modal  for  the  photometric  misregistration 
function*  we  oan  oheose  areas  In  the  two  Images  which  are 
assumed  to  be  photometrically  equivalent  and  eolve  for  a 
oomblned  gain  and  offset  error  which  will  minimize  the 
dlfferenoe  between  the  areas  using  conventional  least 
squares  techniques,  Chapter  V  describes  photometric 


Cl]  JPl  has  prooessed  the  Mariner  '69  Images  to  reduce  these 
errors,  Non-uniform  vldlcon  response  Is  corrected  from 
extensive  calibration  tables  for  the  vldlcon,  Residua) 
Images  are  the  contributions  of  previous  Images  to  the 
vldloon  output  for  a  given  Image  <l,e,  the  vldlcon  surface 
has  a  memory),  Residual  Images  are  partially  removed  using 
calibration  tables  and  previous  Images, 
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rag  I  strat Ion  In  data  I  I , 

DIFFERENCE  ANALYSIS!  Onca  two  Images  are  both 
geometr | oa  I  |y  and  ohotometr I ea I  I y  aligned,  the  analysis  of 
differences  between  the  Images  can  begin,  Certain  classes 
of  known  differences  are  expected,  These  Include  albedo 
differences  due  to  errors  In  the  photometrlo  model  of  the 
planet,  variations  of  the  photometric  function  from  p|aoe  to 
Place  on  the  planet,  and  errors  due  to  the  effects  of  slopes 
(such  as  orater  r I m s )  on  the  photometric  function, 

Given  Images  wn I  oh  were  taken  w|th  approximately  the 
same  I  I  I um | nat | on»  view  and  phase  angles,  the  above  sources 
of  error  should  be  minimised,  The  Mariner  '71  Mission  B 
Images  near  per|aps|s  are  Intended  to  satisfy  the  above 
requirements,  and  hence  these  sources  of  error  (except  f0r 
regional  variation  of  the  photometric  function)  can  be 
largely  Ignore-*. 

The  remaining  albedo  dlfferencee  can  be  attributed  to 
temporal  variations  of  the  photometric  function  due  to  such 
phenomena  as  clouds,  dust  storms#  etc, 

Analysis  of  these  albedo  variations  requires  that  the 
pixel  by  Pixel  albedo  differences  be  reduced  to  area 
differences  end  other  graphical  representations,  One  useful 
form  of  area  difference  display  Is  a  drawing  of  albedo 
dlfferenoe  contour  lines,  If  the  albedo  difference  has  a 
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wt||  defined  outline  then  e  drawing  of  this  outline  Is 
usefuli  Another  useful  display  of  differences  Is  a  oraph  of 
the  size  of  an  albedo  difference  area  versus  time  £  f  o  r  a 
fixed  difference  level)  or  a  graph  of  the  magnitude  of  the 
albedo  difference  versus  time  (for  a  fixed  area). 

Chapter  VI  describes  these  difference  analysis 
teohn  |  cues. 

I , C  NOTATION 

To  facilitate  the  printing  of  th | s  dissertation  by  the 
line  Printer  at  the  Artificial  Intelligence  Project,  some 
compromises  In  mathematical  notation  ware  made,  In 
particular,  superscripts  and  subscripts  are  not  possible, 
and  many  standard  mathematical  symbols  do  not  exist, 
Consequently,  the  scholarly  use  of  an  abundant  oolleotlon  of 
Greek  letters  other  than  ®.  I3.  and  t  w|l|  not  bo  found 
here , 

ZERO  1  Zero  Is  printed  with  a  s|asn  through  It: 

zero  s  0 

EXPONENTS!  Exponents  are  denoted  by  the  ALGOL  40 
notation,  A  to  the  power  N  Is  written  as: 

A  »N , 


15 


INDICES!  Indie®*  of  vectors  and  matrices  art  written  In 
ths  ALGOL  60  notation  as: 

AC  |  *  J]  ,  V C k ] 

VECTORS:  Vsc'.ops  whloh  ara  formed  from  a  collection  of 
scalar  expressions  are  surrounded  by  parentheses: 

{ U, V  )  «  <  x ,  y,  z) 

When  It  |s  necessary  to  specify  a  column  veotor.  the 
transpose  notation  Is  used: 

<  x#  y, z) ' 

MATRICES:  Matrices  which  are  formed  from  a  collection 
of  scalars  expressions  are  written  using  as  many  lines  as 
there  are  rows  In  the  matrix.  Thus,  the  3x3  Identity  matrix 
would  be  written: 


I  s 


10  0 
0  10 
0  0  1 


The  transpose  of  a  matrix  M  |9  written* 


transpose  <M)  ■  M' 


The  Inverse  of  matrix  M  is  written* 


lnv<M) 
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INNER  PRODUCT!  Th*  Inner  produot  of  two  vectors  VI  and  V2  Is 
wr I tten: 

<V1,V2> 

EXPRESSIONS!  Arithmetic  operations  between  scalar*, 
veotors  end  matrices  are  denoted  In  conventional 
mathematical  notation  whenever  possible,  Consequently,  a 
particular  symbol,  suoh  as  •  ,  can  mean  different  operations 
depending  upon  the  context, 

EUCLIDEAN  NORM;  The  length  of  a  vector  V  (Euclidean 
norm)  Is  written! 

1  I  V  I  I 

INTEGER  FUNCTION i  The  Integer  part  of  a  real  number  X 
(|,e,  the  greatest  Integer  not  exceeding  X)  Is  written! 

exp 

SUMMATION!  The  sum  of  an  expression  over  an  Index 
variable  Is  written  using  two  lines!  the  first  containing 
the  word  "sum”  followed  by  the  expression,  and  the  second 
line  containing  the  name  and  range  of  the  Indew  of  summation 
under  the  word  "sum",  Using  this  notation,  the  product  of 
two  matrices  A  and  B  would  be  written! 

CC I ,  J  ]  «  sum  AC  l,kJeBCk,  J'D 

l<kSn 

INTEGRATION!  The  Integral  of  an  expression  Is  written 


analogously  to  a  sum,  The  Integral  of  funotlon  f(x)  ovar 
tha  ranga  <-l#l)  Is  wrltteni 

Intagral  f(x> 


1,0  vocabulary 

Soma  of  the  words  used  |rt  tn|s  dissertation  are 
unoommon#  soma  ara  localisms#  and  soma  are  highly  teohn|ca|, 
Tharafora#  some  such  words  ara  defined! 

albedo  -  The  ratio  of  the  reflected  light  to  tha  Incident 
light  normal  to  a  surface.  A  white  Lambertian 

surfaca#  normel  to  tha  Inoldent  sunlight#  would  have 
an  albedo  of  one# 

apoapsls  -  apogee  -  The  point  In  tha  orbit  most  distant  from 
the  planet. 

oerlepsls  -  perigee-  The  point  In  the  orbit  closest  to  tie 
Planet. 

pixel  -  Abbreviation  for  "cloture  element"#  referring  to  the 
light  value  at  a  point  In  a  Ploture, 

The  next  four  ohapters  describe  |n  detail  the 
techniques  for  normalizing  and  registering  Images# 


CHAPTER  II 

GEOMETRIC  IMAGE  NORMALIZATION  BY  "DEAD  RECKONING" 

Image  normalization  by  "dead  reckoning"  refers  to 
taehnlauaa  which  g jometr I ea  I  I  y  and  ohotomatr | oa I  I y  register 
Images  using  only  calibration  Information  suoh  as  camera 
ooaltlop  and  or|entat|on»  oamera  g«n*lt|v|ty,  eto,  rather 
than  Information  contained  within  the  Imagaa  themae|ve», 
The  quality  of  such  normalization  techniques  la  determined 
by  the  quality  of  the  calibration  Information,  For  Mariner 
6  and  7,  errors  In  tha  calibration  result  In  a  geometric 
m | areg I atrat I  on  o?  5  to  10  pixels  <25  to  50  kilometers  on 
tha  surface), 

PRQBLEMi  Given  two  Images  taken  from  different 
spacecraft  (camera)  positions  and  or lentatlons#  possibly 
from  different  spacecraft  (cameras)*  geometrically  transform 
the  region  In  each  Image  which  Is  common  In  the  soene  (on 
the  surface  of  the  planet)  so  that  the  two  Images  can  be 
oomps  red  pixel  by  o  I  xe I , 


DEFINITION!  An  iotfli  |s  a  raa|  funot|on  of  two  varlablas 
F(u,v)  representing  tha  light  tntanalty  racalvad  at  point 
(u#v)  on  tha  I  mag#  plana  of  some  Imaging  davlca  (eg,  a 
oama ra ) , 

All  of  tha  I maga  acannlng  ayatama  of  Intaraati  however, 
ara  dlacrata  ayatama  whleh  quantize  the  light  level  at  a 
fixed  number  Cog,  512)  of  Intenalty  levels  over  a  fixed 
rectangular  array  of  points  (eg,  945  x  702), 

Such  discrete  Imaging  systems,  furthermore,  encode  the 
light  level  Integrated  over  an  area  rather  than  at  a  point, 
This  Integration  can  be  formalized  ael 

F(x,y)  «  Integral  (  f<u#v)  *  g(x«u,y-v> > 

“-<u, v<+- 

whare  f  Is  the  Intensity  funotlon  at  a  point  and  g  Is  the 
•’point  spread"  funotlon  of  tha  Imaging  devloe  (>"*osenfe|d 
[  1969  J ,  p  44),  which  Is  usually  adjusted  by  cptlca|  or 
alactronlo  defooualno  to  minimize  the  errors  duo  to  aliasing 
Introduoed  by  discrete  sampling, 

In  terms  of  scene  coordinates,  there  exist  two 
geometrio  transformations  T1  and  T2  which  map  coordinates  of 
oolnts  In  the  Images  II  and  12  Into  scene  coordinates,  suoh 
that  when  <ul,vl)  and  (u2,v2)  represent  the  same  point 
<x,y»z)  In  the  scene,  tha  following  Is  true! 
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Tl(ul.vl)  3  U,y,z>  ■  T2(u2,v2) 


(2,1) 


The  next  sections  derive  tranaf ormat I one  T1  and  T2 
using  geometric  models  for  ths  sosns  and  camera, 

II, A  GEOMETRIC  rODELS 

PROJECTIONS  OF  SCENESi  A  orojeotlon  of  a  soana  Is  some 
gaomatrlo  mapping  T(x,y,z)  whloh  maps  tha  eoordlnataa  of 
points  In  tha  3-0  soana  Into  coordinates  of  the  2-D 
orojactad  Image,  Soma  polnta  In  tha  soana  do  not  map  to  the 
orojaotad  Image  (a,g.  points  on  tha  opposite  side  of  an 
object  being  viewed  by  the  aye), 

Tha  moat  fam||far  projection  Is  the  perspective 
projection  (Fig,  2.1a),  which  maps  the  point  Ps  In  the  3-D 
scene  Into  tha  point  PI  whloh  Is  the  Intersection  of  the 
Image  plana  and  the  straight  line  through  the  lens  canter  0 
and  Ps,  If  there  Is  any  other  point  Pa  on  the  line  between 
0  and  Ps,  than  Ps  Is  said  to  be  "occluded”  by  Pa,  Occluded 
Dolrts  are  not  mapped  to  the  Image  plane. 

Another  useful  projection  is  the  orthographic 
projection  (Fig,  2-lb)  whloh  maps  th»  point  Ps  in  the  3-D 
scene  Into  the  point  PI  sueh  that  Ps  !s  on  the  line  normal 
to  the  Image  p I  ana  at  P 1 , 

camera  MODEL i  The  camera  can  be  geometr | oa I | y  modeled 
as  a  lens  center  and  an  Image  plane  (Fig,  2-la),  At  each 
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IHAug  PLANE 


Figure  2-lc . 


Inverse  Perspective  Projection 


point  (u#v) 

on  the 

Image  plane# 

the 

camera  records 

the 

amount  of 

1  1  oh  t 

received  from 

the 

scene  along  the 

1  1  ne 

through  the  lens  center  and  point  <u#v>,  Thus#  the  camera 
Is  said  to  generate  a  oarspsotlvs  Imags  of  tha  soene,  In 
ordar  to  normalize  Images  taken  from  different  camera 
posltlone#  |t  | s  necessary  to  determine  the  soene 
coordinates  of  points  In  the  perepeot|ve  Images# 

SCENE  MODELS j  Theoretically#  general  scenes  are  easy  to 
model  gaometr I oa |  I y>  but  In  practice  this  oan  be  very 
difficult  to  accomplish,  A  very  general  class  of  scenes  may 
be  modeled  by  aseumlng  that  the  scene  Is  made  from  objects 
whose  surfaoae  are  ooaoue  and  era  deecrlbad  by  a  function 
H(x#y#z)«0,  A  soene  consisting  of  a  sphere  of  radius  r 
centered  at  tha  origin  can  thus  be  modeled: 

H(x#y#z)  ■  xt2  ♦  yt2  *  z*2  -  r»2  *  0  (2,2) 

Tha  planet  Mars  can  be  approximated  by  an  oblate  spheroid 
with  eauatorlal  radius  Req*3393,4  km  (equator  In  the  x-y 
Plane#  with  z«0)  and  polar  radius  RpoI*3375,6  km  (poles 
along  the  z>axls),  with  this  model#  H  becomes: 

H ( x # y #  z )  ■  x»2  +  y ♦ 2  *  ( Z*Reo/Rpo I ) ♦ 2  -  Req*2,  (2,*) 
II, B  THE  INVERSE  PERSPECTIVE  PROJECTION 

Inverse  projeetlone  map  coordinates  of  points  In  Images 
which  are  projections  of  scenes#  back  Into  coordinates  In 


the  scene,  Geometric  models  fop  the  scene  end  the 

projection  ere  required  In  order  to  Invert  the  projection, 

The  Inverse  perspeotlve  Image  projection  Is  the 
projection  from  two  dimensional  perspeotlve  Image 
coordinates  to  threa  dimensional  spatial  coordinates,  This 
projection  maos  Point  <g,v>  In  the  perspective  Image  p  ane 
to  the  olosest  point  (x,y,z)  In  the  3*0  scene  which  Is  along 
the  line  through  the  lens  oenter  and  <u*v),  Any  point  along 
that  line  Is  given  |n  camera  relative  coordinates  (the  lens 
center  Is  the  origin*  with  axes  u»v,  and  w>  as  a  funotlon  of 
the  depth  parameter  d  as  d*(u*v,n,  where  f  Is  the  Image 
dlstanoe  of  the  oamera  (F|g,  2-lc>, 

To  traneform  camera  relative  coordinates  Into  scene 
relative  coordinates*  a  linear  transformation  L  Is  doflnedi 

L ( u  *  v , w ) )  ■  R(u,v,w>'  +  Pc  (2,4) 

where  I 

1)  R  le  a  rotation  matrix  from  camera  spaoe  orientation  to 

ecane  orientation,  Appendix  A  derives  matrix  R  from 
geometrlo  calibration  data, 

2)  Pc  Is  the  ooeltion  of  the  camera  relative  to  the 

origin  of  the  scene  coordinate  system, 

Using  these  definitions*  the  Inverse  perspective 
projection  T  of  point  <Uiv>  !•  defined  In  scene  coordinates 
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by  finding  the  unknown  depth  parameter  d  suoh  that* 


T ( u#  v )  ■  L< d*(u#  v»  f ) ' )  <2,5a) 

■  R(d*(UiVif )' )  +  Pc  (2,5b) 

*  d*R( u#  v»  f ) '  ♦  Pc  (2,5c) 

such  that  d>0,  d « m I n *  and  (2,6  ) 

0  *  H  (  T(u#v)  )  (2.7a) 

»  H  (  d*R( ui Vi  f  > '  ♦  Pc  )  (2,7b) 


where  tha  primed  vectors  denote  oo|umn  vectors,  Equation 
(2,5)  requires  that  T(UiV>  be  a  perspective  view  of  the 
scene  from  a  camera  at  position  Po  wltn  orientation  defined 
by  R  and  with  Image  distance  f,  Equation  (2,7a)  requires 
that  T ( u  #  v )  be  a  point  In  the  scene,  and  (2,6)  requires  that 
It  be  the  point  closest  to  the  camera, 

Using  the  above  definitional  the  perspective  projection 
T 1  Is  the  Inverse  of  T  and  can  be  defined  as  follows: 

Let  the  transformation  T”  be  defined* 

T " ( x i y , z )  s  |nv(R)*<txiyiZ)'-Pc>  (2,8) 

where  |nv(R)  Is  the  Inverse  of  matrix  R,  Note  the 
f  o ! lowing: 

T " ( T ( u i v ) )  *  T”(R(d*(u#v,f )')  +  Pc)  (2,9a) 

*  I nv ( R ) • ( R ( d* ( ui v » f ) '  )  -  Pc  ♦  Pc)  (2,9b) 
=  lnv(R)*(R(d#(u»v,f ) '  ) )  (2,9c) 
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d*( u#  Vi f  )  ' 


(2,9d) 


If 


(ui v#w) 1  •  TM(x»y#z) 


<2, 10*} 


than*  tha  parspactlva  projaotjon  T'  la  daflnad  aai 


T' <  x#  y#  z)  •  f  * 


(ill  v) 


(2,10b) 


Nota  that  w«d*f( 


II, C  SPECIAL  CASES  OP  THE  INVERSE  PERSPECTIVE  PROJECTION 

Eduatlona  (2, 5-2,7)  art  In  ganaral  vary  difficult  to 
ao|v*  bacaus'j  of  tha  oomplaxlty  of  H,  This  saotlon  darlvas 
thalr  solution  for  tha  spaclal  oaaas  of  apharlcal  and 
spharc'dal  soana  modaja, 

SOLUTION  FOR  A  SPHERE  I  Tha  Invarsa  psrapectlva 
transformation  problam  can  ba  aaally  solved  for  known  camara 
position  and  orlantatlon  with  raspaot  to  a  sphare,  In 
particular*  for  a  sphara  of  radius  r  oantsrad  at  tha  soans 
origin,  wa  hava  for  a  g|van  point  P  on  tha  spharai 

|  |  P  |  |  *  r,  |,a,  Pxt2«-Py»2+Pzt2  •  rt  2,  (2,11) 

So*  glvtn  tha  parspactlva  projaot|on  of  P  In  Imaga 

ooordlnatas  (u*v),  w*  know  thati 
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0  *  I  I T ( u»  v ) | | *2  •  p* 2  (2,12a) 

■  I  |R(d*(u#v»f >' )*Pol | *2  -  r*2  (2,12b) 

■  I | R | 1*2*1 Ida(uiv.f)' I |*2  (2,12c) 

*  2*<R(d*(u» v»f >' ),Pc> 

*  I  I Po I  1*2  -  r*2 

*  d*2  *  | | (u»v»f ) | | *2  ( 2 , 12d ) 

*  2*d  *  <R(u#v,f)',Pc> 

*  I i Pc  I  1*2  -  r  *2 

d  Is  found  by  solving  ths  quadratic  equation* 


d  »  (-b  -  sqrt(b*2-4#a*c> >/(2*a> 


whirs 

•  B  I  I <u»v»f ) | |*2  »  u*2*v»2*f *2 
b  *  2*<R ( u»  v»  f ) ' » Pc> 
o  *  I  I  Pc  I  l*2-r*2 


(2,13) 


(2,14a) 

(2,14b) 

(2,14c) 


Thi  smaller  solution  <-  sqrt)  of  th»  quadratic  equation  Is 
ths  only  moan  I ngf  u I  on#  s|not  ths  point  on  the  sphere 
corresponding  tc  the  larger  solution  Is  at  a  greater 
distance  from  the  camera  and  Is  therefore  occluded  by  the 
point  corresponding  to  the  smaller  solution,  Complex 
solutions  correspond  to  earners  rays  which  do  nut  Intersect 
the  sphere, 


SOLUTION  FOR  An  SPHEROID*  The  solution  for  a  sphere 
easily  generalizes  to  a  soherold  by  Introducing  an 


eccentricity  constant  a  along  tha  z-ax|s  (North  to  South 
pola  axle)  auoh  that! 

x*2  *  y ♦ 2  ♦  <e*z)*2  ■  r*2  (2,13) 

If  tha  aocantrlclty  matrix  E  I*  defined  as> 

10  0 

E  i  0  1  0  (2,16) 

0  0a 

than  d  Is  found  by  solving  tha  following  quadratlo  aauatlonJ 

0  ■  | | E«T ( u#  v ) | I *2  -  r *2  (2,17a) 

a  | |E«(R(d*(uiV»f >  >  *Pe | |t2  -  r*2  (2,17b) 

a  d*2  •  I |E*R(u*v»f >  |  I *2  (2,17c) 

♦2«d#<E*R(u#v,f),E(Pe>> 

+| | E ( Pc ) I | *2  •  r f2 

II, D  EXAMPLES  OF  GEOMETRIC  NORMALIZATION 

This  sactlon  contains  axamolas  of  tha  gaomatrlc 
normalization  techniques  apollad  to  f ar-encounter  Marlnar  7 
Dlcturas  of  Mars,  Thaaa  olcturas  wars  takan  when  the 
spacecraft  was  sufficiently  far  from  Mars  to  sea  tha  entire 
disk  of  the  planet, 

Figure  2-3  shows  the  disk  of  the  planet  (the  clrola) 
with  the  north  oole  at  the  too  as  seen  from  an  orthographic 
projection  In  tha  direction  0  latitude*  0  longitude  (I, a, 
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(0  lat,,  0  long,)  Is  the  oentor  of  the  olrcle),  Th§  dotted 
linos  surrounding  tho  elrolo  Indicate  tho  outllnos  of  two 
oloturo*  whJoh  ops  doslgnotod  7F75  and  7F78,  Tho  vectors  In 
the  circle  Indlcato  tho  dlpoetlon  to  the  spacecraft  and  the 
eun  frorr  the  oentpa|  points  of  tho  two  olcturoo,  Tho  square 
Indloatos  tho  aroa  on  tho  aurfaoo  for  which  an  orthographic 
orojootion  of  eaoh  ploturo  will  bo  gonorated. 

Figure  2-4*  shows  tho  aroa  of  tho  ploturo  7F75  (the 
dotted  area)  which  Is  spoolflod  by  the  square  In  figure  2-3, 
Tho  reotang|e  Is  the  943x702  pixel  outline  of  oloturo  7F75, 
Similarly,  figure  2-4b  shews  the  aroa  of  7F78  which  Is  to  be 
or thograph 1 ca I  I y  projected, 

Figures  2-5*  and  2-5b  show  the  pictures  7F75  and  7F78, 
as  soon  from  tho  spacecraft,  Note  that  tho  orator  (Nix 
Olymeloa),  which  Is  in  tho  upper  middle  p*rt  of  7F75,  Is  In 
tho  upper  right  part  of  7F76,  This  Is  because  the  Planet 
rotated  by  34  dogre«s  between  the  two  olotures,  A  scale 
factor  difference  Is  also  obvious, 

Figures  2-6a  and  2-6b  show  the  orthographic  Projection 
of  eaoh  Image  as  soeelfled  by  figures  2-3,  2-4*,  and  2-4b, 
In  these  normalized  pictures,  the  crater  looks  about  the 
same,  and  various  features  match,  However,  there  Is  a 
definite  geometric  error  In  the  registration  of  these 
I  mages , 
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CHAPTER  Ill 


ACCURATE  GEOMETRIC  REGISTRATION  OF  NORMALIZED  IMAGES 

III.A  FORMAL  DEFINITION  OF  GEOMETRIC  MISREGISTRATION 

Let  ue  suppose  that  we  have  two  gspmstr I oa | I y 
normal lz«d  Images  (F1,T1)  and  (F2,T2)  where  the  T|  are 
tranef ormat | ons  from  Image  coordinates  to  actual  3-spaoe 
ooord Inateeii  rather  than  those  oredloted  by  Chapter  II, 
which  oontaln  errors  due  to  the  oamera  and  soene  models, 
The  FI  are  the  Intensity  funot|one  |n  the  Images,  The 
Images  are  eald  to  be  "geometr I oa | I y  ml sreg I  stared"  at  point 
( u>  v )  I  f  * 

Tl(u,v)  P  T2 ( ui v )  (3,1) 

We  will  dvjflne  two  functions  du<u»v>  and  dv(u»v)  whloh 
represent  the  m|eal)gnment  of  two  Images  In  the  u  and  v 
directions  respectively  as  a  funotlon  of  the  poeltlon  In  the 
Images  ael 

<du<u# v) #dv(u» v) )  ■  <(du#dv) iTi(u»v)"T2(u+du>v+dv>)  (3,2) 
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F  I Oure  2-5a 
7F  75  Original 


Figure  2-5b 
7F78  Original 


Figure  2-6ft 
7F  75  \nrma I  I  zed 


Figure  2-6b 
7F78  Normalized 


These  misalignment  functions  te||  us  that»  given  t  point 
(u,v>  In  Imags  1,  this  sam»  point  |n  ths  sosns  Is  at 
(u+du#v+dv)  In  Image  2, 

If  we  have  tha  mleallgnment  funotlons  daflnad  at  avary 
point  In  tha  dlsorata  Images,  than  ws  ean  daf|na  a  naw  Imaga 
(F3, T3 )  whloh  la  exactly  rsglstered  with  Imaga  1  as  follows: 

F3(u*v)  *  F2(u*du(u» v)»v*dv(u>v) )  (3,3) 

(assuming  an  adaousta  modal  for  F2  patwaen  data  points) 

T3(u*v)  *  T2( u*du( Ui v> i v+dv(u# v) >  (3,4) 

*  Tl(u»v> 

Tha  naxt  sections  empirically  derive  tha  misalignment 
funotlons  for  a  pair  of  Images? 

!  1 1 1 B  MODEL  FOR  MISREGISTRATION 

Because  of  geometric  errors  primarily  In  tha  Mariner 
'69  camera  modal  (camera  position  and  orientation),  Images 
which  have  been  geometrically  normalized  by  "dead  reokonlng" 
era  not  exactly  registered,  It  can  be  shown  that  small 
errors  In  the  eamera  modal  cause  primarily  a  translational 
misalignment  between  geometr I oa | |y  normalized  Images,  There 
are  also  small  rotational  and  scale  factor  errors,  Higher 
order  errors  might  exist  due  to  residual  distortions  In  the 
optles  and  errors  In  the  3*0  model  for  the  scene, 


On*  might  observe  th»t  fl|v*n  two  Images  which  are 
approximately  aligned,  people  very  easily  match  most 
feature*  (such  as  oraters)  which  correspond,  However,  the 
mechanisms  used  |n  Image  matching  by  people  are  not 
neoessarlly  those  best  for  automatlo  Image  alignment  by 
computer , 

Th*  predominantly  translational  nature  of  the 
misalignment  of  Images  suggests  that,  when  local  areas  In 
Images  are  appropriately  translated,  the  Images  will  match, 
If  the  two  Images  were  taken  under  sufficiently  sl:,i||ar 
photometric  conditions  (such  as  Illumination,  oolor  filters, 
etc,)  then  the  Quality  of  the  match  can  be  measured  by 
statlstloal  cross  Correlation,  Th*  registration  technique 
which  has  been  developed  |s  to  maximize  th*  local  cross 
correlation  of  the  Images  as  a  function  of  the  translation 
of  one  Image  with  respect  to  the  other, 

An  alternative  procedure  would  be  to  search  one  Image 
for  a  distinct  feature  such  as  a  crater  or  some  other 
topoQraphlc  feature,  and  then  searoh  tne  second  Image  In  a 
predicted  area  for  th*  same  feature,  Such  feature 
recognition,  however*  reoulres  that  each  Image  which  Is  to 
be  aligned  oontaln  th*  neoessary  number  (and  Quality)  of  the 
types  of  features  which  can  be  r*oogn|zed,  Experience  with 
a  crater  finder  C13  shows  that  In  order  to  reliably  locate 
orators,  the  contrast  of  th*  crater  r|ms  must  be  good,  and 
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that  the  s I gna I  - to-no I sa  ratio  must  d#  high,  Evan  whan 
thaaa  conditions  ara  mat#  tha  cross  oorra|atlon  allonmant 
procedure  ssama  to  ba  superior#  bacauaa  of  Its  relative 
I  naans  1 1 1 v I ty  to  different  types  of  terrain  and  to  noise, 
Thera  also  appears  to  bs  no  Inherent  speed  advantage  for  a 
feature  deteotor,  One  outstanding  virtue  of  a  orater  finder 
would  be  the  ability  tr  I'joats  craters  and  develop  a  3-D 
depth  modal  so  that  the  photometric  errors  due  to  crater 
slonas  could  be  removed,  Suoh  slopes#  however#  cause 
significant  errors  on|y  when  the  Illumination  and  viewing 
ancles  are  quite  different  between  the  pictures  being 
rag  I atared, 

Cross  correlation  maximization  was  selected  for 
geometric  alignment,  beoause  of  Its  simplicity  and 
applicability  to  varying  terrains, 


C13  A  crater  finder  was  developed  wh|oh  looks  for  h|gh 
oontrast  edges  wh|C'  contain  large  circular  arcs,  Many 
craters  are  recognized  by  the  fraction  of  th« I r  rim  which  |s 
a  high  oontraut  crescent.  Th#  outside  curv#  of  the  cresoent 
Is  a  clroular  aro  with  approximately  the  same  radius  «nd 
center  ae  the  orater, 
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IIl.C  CROSS  CORRELATION  AS  A  MEASURE  OF  THE  QUALITY  OF 
REGISTRATION 

Correlation  Is  a  statistical  measure  of  agreement 
between  two  functions,  For  two  random  variables  >  and  Y 
defined  on  a  discrete  set  of  n  points*  their  normalized 
oross  correlation  <cor>  Is  defined  In  terms  of  the  means  (E) 
and  standard  deviations  <  s  d  >  asi 

E ( X*Y )  -  E(X>»E(Y) 

oor  I X,  Y )  ■  . .  (3,5) 

sd(X>  •  sd<Y> 

where 

E  ( X  >  ■  sum  <  X  C I  3  >  /  n  (3,6) 

lSISn 

and 

Sd(X)  *  sort  <  E(Xt2>  -  E(X>*2  )  (3,7) 

To  apply  oross  correlation  to  an  area  of  two  Images 
with  Intensity  functions  FI  and  F2,  the  one  dimensional  sum 
In  eouatlon  (3,6)  becomes  two  dlmenslonali  and  we  make  the 
following  definitions  for  the  variables  X  and  \,  Given 
( u, v )  as  the  center  of  a  ( 2n+l ) x ( 2n*l )  area  to  be  oorre|ated 
and  a  (du»dv)  translation  veotor,  define  X  and  Y  asi 

XC I  *  J3  *  Fl( l+u*n, J+v-n>  0SI»JS2n  (3,8a) 

Y C I »  J 3  *  F2( I ♦u*n4du# J^v*n*dv )  (3,8b) 

The  normalized  correlation  funotlon  has  the  property 
that  It  does  not  vary  with  the  size  of  the  correlation  area, 
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or  with  the  moons  »nd  standard  dov|otlons  of  X  and  Y, 
Subjoctlvoly  this  guorontoos  that  tho  correlation  function 
Is  not  affootod  by  tho  avorago  |ntons|ty  | ova |  and  |ovo|  of 
oontrast  In  tho  window,  Tho  duality  of  tho  corrolatlon 
moasuro  Is,  however,  strongly  rolatod  to  s I gna I -to-no | so 
ratio,  Inoroasod  nolso  will  Inoroaso  tho  standard 
dovlatlons  In  oauatjon  <3, 5)  without  Incroaslng  tho  product 
term  E(X*Y), 

Th I o  offoct  con  bo  dorlvad  by  assuming  that  tho  two 
windows  aro  Identical  except  for  uncorrolatad  nolso,  Thon 
thoy  can  bo  roprosontodl 

X  *  Z  ♦  N1  ,  Y  «  2  ♦  N2  (3,9a) 

Hhoro  Z  Is  tho  "true"  valuo  of  window,  If  Ni  end  N2  are 
unoorre |atod  nolso,  and  have  tho  samo  standard  deviations, 
thon,  tho  cross  oorro|atlon  of  X  and  Y  Isi 

E ( ( Z*N1 ) * <  Z+N2 ) )  -  E ( Z  +  Nl )»E ( Z*N2 ) 


0or  ( X,  Y )  . - . .  (3,9b) 

od( H+Nl )  *  sd(Z*N2) 

E(Z*Z)  -  E(HMECZ) 

8  . : .  (3,90 

sd ( Z*N1 )  •  sd(Z+N2) 

a  var ( Z )/var < Z+N)  (3,9d) 

*  1  -  var(N)/var (Z+N)  (3,9e) 


whoro  var  Is  the  varlanoe,  High  uncorrelated  noise  levels 
oonsoquently  w|||  deeroaso  tho  valuo  of  tho  corrolatlon 


function,  This  suggests  thst  the  correlation  function 
should  be  evaluated  only  |n  areas  where  the  standard 
deviations  of  the  windows  are  significantly  greater  than  the 
standard  deviation  of  the  noise,  and  that  the  search  for 
maximum  correlation  should  f|nd  a  correlation  value 
approximately  equal  to  the  value  of  equation  (3,9e>, 

For  a  desired  level  of  correlation  quality*  the  window 
size  oan  be  adjusted  according  to  the  local  standard 
deviations  of  the  two  Images,  Increasing  the  window  size 
(l,e.  Inoreaslng  n)  exoesslvely  w!|l  cause  poor  results  due 
to  non-trans I  at | ona |  (rotation  and  seals  factor)  distortions 
between  the  two  Images  (Appendix  B), 

I J I  I o  local  cross  correlation  maximization  TO  DETERMINE 
LOCAL  MISREGISTRATION 

Analyses  of  the  errors  In  the  Mariner  6  and  7  geometric 
model  show  that  Images  which  are  normalized  by  ’’dead 
reckoning"  should  contain  primarily  translational  errors  In 
alignment  with  smaller  rotational  and  scale  factor  errors, 
The  results  In  tables  and  figures  3-1  to  3-7  oonflrm  this 
urealctlon*  and  snow  that  constant  translational  errors 
domlnute  other  errors  by  a  factor  of  more  than  10)1,  The 
presenoe  of  non-t rans | at  I ona |  errors  outs  an  upper  limit  on 
the  size  of  the  ( 2n  +  l )  x (2n+l )  correlation  window, 


Therefore,  to  align  two  Images  using  correlation,  It  fs 
sufflclant  to  find  the  (duidv)  translation  of  ona  Image  with 
raspact  to  tha  othar  whleh  maximizes  tha  value  of  the 
oorrelatlon  funotlon  around  eaoh  point  <u»v)  In  tha  Images, 
This  (du*dv)  translation  Is  oal|sd  the  "misalignment  veotor" 
for  tha  area  of  correlation  oentered  at  <u#v>,  It  has  not 
been  nacessary  to  maximize  the  oorrelatlon  with  respect  to 
other  parameters  such  as  rotation  and  scale  faotor  changes, 
because  the  magnitudes  of  these  errors  are  small  relative  to 
translational  errors,  and  It  |s  possible  to  find  the  local 
correlation  maximum  using  only  translations,  These  higher 
order  errors  a re  found  whan  many  different  misalignment 
vectors  are  modeled  over  the  entire  Image  as  described  In 
III.E, 

SEARCH  TOR  MAXIMUM  -  STRATEGIES!  The  subjeot  of  this 
section  |s  to  describe  two  different  strategies  to  search 
for  the  translation  vector  which  Produces  the  maximum 
oorrelatlon  between  two  Images,  The  predominant  oonstant 
translational  misalignment  suggests  that  the  search  for 
maximum  oorrelatlon  should  be  oonflned  to  the  neighborhood 
of  that  predominant  error  vector. 

The  primary  reason  for  considering  these  search 
strategies  Is  to  reduce  the  number  of  evaluations  of  the 
correlation  function  and  thus  Improve  the  performance  of  the 
Image  registration  system, 
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The  alignment  strategies  erei 


1)  Global  BtrateQyJ  determ|nt  the  predominant  global 

m|3al|gnm#nt  of  tha  two  Imagesi 

2)  Local  strategy?  determine  the  |ooa|  misalignments  of  the 

two  Images  by  searching  In  limited  neighborhoods 
predicted  by  tha  global  m | ea I  I gnment  model, 

GLOBAL  SEARCH  STRATEGY?  The  purpose  of  the  global 
soarch  strategy  Is  to  search  for  the  translation  vector 
which  produces  tha  maximum  correlation  between  two  Images 
without  knowledge  to  limit  the  search  to  a  small 
neighborhood,  Because  the  correlation  function  Is  evaluated 
only  at  Integer  values  of  the  <du,dv)  displacement  vector* 
the  global  searoh  strategy  oould  be  Implemented  by 
exhaustive  evaluation  of  the  correlation  function  at  all 
Integer  translations,  However*  some  analysis  of  the 
correlation  function,  and  the  areas  of  the  Images  being 
correlated*  shows  that  one  oan  limit  the  number  of 
evaluations  of  the  correlation  function  cons  I  derab | y , 

If  one  knows  the  sharoness  of  the  correlation  peak  and 
Its  amplitude  relative  to  other  relative  peaks*  then  one  can 
determine  a  eearch  grid  spacing  which  will  guarantee  finding 
the  absolute  peak,  If  for  Instanoe*  the  absolute  peak  |$  k 
units  wide  at  the  level  of  any  other  relative  peak*  then  the 
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global  starch  can  be  carried  out  on  a  kxk  grid*  reducing  the 
number  of  evaluations  of  the  correlation  function  by  a 
factor  of  k*2  (see  figure  3-0a),  Various  Parameters  (such 
as  window  size*  spatial  averaging*  etc*)  can  be  adjusted  to 
oroaden  the  peak  sufficiently  to  allow  #  |  e  x  I  b  1  e  choice  of 
the  grid  spacing  (Mgs,  3-0o  and  3-0d  show  the  effects  of 
spatial  filtering  on  correlation)* 

If  one  wants  to  search  for  the  correlation  maximum  by 
evaluating  the  correlation  function  on|y  at  points  on  a  kxk 
pixel  grid*  then  the  sampling  theorem  says  that  one  should 
limit  the  frequency  spectrum  of  the  correlation  function  to 
spatial  wavelengths  longer  than  2k  pixels,  This  Is 
successful  only  |f  the  power  speotra  of  the  two  Images  have 
sufficient  power  at  spatial  wavelengths  longer  than  2k 
pixels.  The  disadvantage  of  low  pass  spatial  filtering  Is 
that  high  spctlal  frequency  Information,  which  usually  best 
characterizes  topographic  features,  Is  lost,  Therefore,  a 
searoh  for  the  local  correlation  maximum  of  unflltered 
Images  Is  desirable  In  a  limited  nelghoorhood  around  the  low 
frequency  maximum, 

LOCAL  SEARCH  STRATEGY  l  When  the  predominant 
translational  misalignment  between  two  Images  Is  known,  the 
searoh  for  the  local  misalignment  can  be  limited  to  a  small 
neighborhood*  thus  limiting  the  number  of  evaluations  of  the 
correlation  funotlon,  The  local  search  strategy  begins 
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searohlng  for  maximum  oorrs|atlon  starting  with  a 
translation  vaotor  which  !a  predicted  oy  a  m I srag I strat I  on 
modal  baaed  on  tha  predoM'nent  global  m I srag I strat I  on  vaotor 
and  pravlous  local  rag  I  at  rat J on  veotors,  Tha  stratagy 
saarchas  a  limited  1 1  stan.oa  In  6  dlffarant  dlraotlons 
looking  for  a  maximum,  A  h|||-ol|mb  search  than  follows# 
until  a  (duidv)  point  Is  found  suoh  that  all  8  neighbors 
give  a  lower  correlation  value# 

modeling  the  correlation  function  at  non-integral 

TRANSLATIONS!  Given  the  values  of  the  oorre|atlon  function 
at  a  discrete  sat  of  Integer  translations#  It  is  useful  to 
modal  the  function  at  non-integral  translations  using 
Interpolation  or,  oqu I va I  ant  I y #  fitting  linear  combinations 
of  functions  (such  is  polynomials*  to  the  data# 

In  partloular#  one  oan  f I t  a  polynomial  In  2  variables 
to  the  correlation  function  at  known  translations#  and  solve 
for  the  maximum  of  the  polynomial#  The  method  chosen  was  to 
use  a  least  sauares  f 1 t  of  an  Nth  order  polynomial  In  2 
varlablee#  and  solve  for  Its  maximum  using  a  2-d I  mens | ona I 
generalization  of  Newton's  msthod  (Kowallk  C1968]  pp, 
65-71)  In  the  vicinity  of  the  maximum  emo|r|cal  value  of  the 
-orrelatlon  function# 

Although  It  Is  difficult  to  prove  that  the  correlation 
function  oan  best  be  approximated  In  this  fashion#  there  Is 
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Figure  3-la 
N I  *  o I ymo I ca 


X#c  <0.Y0*il0.0Xx  10, py*  i , con* 

px*  *>.o.pr*  i.ti .cort*  .u 


rigure  3-lb 
Random  Area 


***  7#.r^  ■)0.px*  -j.prs  -i.coh, 
PX*- 1.37.PV--*.»4.COR«:  ,11 


Flgyre  3-lc 
Craters 


<<?■■  ':<f.  >•£*•  70,j"<-  t.pY-  ■' ,  c»  <f? 


F I gure  3-ld 
H I  pass  Craters 


•  ome  evidence  that  |t  canl 

1)  Empirical  evidence*  Jnoreaslng  N,  tha  ordar  of  the 

polynomials  f|t  to  tha  oorralatlon  surfaoe,  produces 
Quintuples  (uCID#vCl3,duCIJ#dvCIJ#oC|])  which  are 
batter  modeled  by  section  III,E  In  terms  of  RMS 
error  (the  cCU  are  the  values  of  the  correlation 
funotion  which  are  used  as  weighting  coefficients 
for  the  least  squares  fit  In  section  JIJ.E), 

2)  Theoretical  evldenpel  The  correlation  function  tends  to 

be  smooth  and  radially  symmetric  about  Its  maximum 
value  (see  figs,  3-0a  to  3-0q),  From  the  Fourier 
analysis  point  of  view#  the  Fourier  transform  of  the 
correlation  surface  Is  the  produot  of  the  Fourier 
transforms  of  the  two  Images  (oomplax  conjugate  of 
one  of  th«m),  Jf  the  two  Images  correlate  well, 
then  the  correlation  surface  Is  approximately 
equivalent  to  the  autooor re  |  at  I  on  surfaoes  of  either 
of  the  Images,  which  tend  to  De  radially  symmetric, 
If  both  Images  have  most  of  their  spectral  power  at 
low  epatlal  frequencies,  then  the  correlation 
eurface  w | I |  also  have  most  of  Its  spectral  power  at 
low  spatial  frequencies,  and  the  correlation  surface 
will  be  correspondingly  smooth  and  broadly  peaked 
around  the  nominal  ml sreg I strat  |  on  vector, 
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EXPLANATION  OF  FIGURES  3-0* , D , e » d l  These  figures  show 
windows  from  oa|rs  of  pictures,  and  cross  saottons  of  the 
oorralatlon  surface  In  tha  v | o I n | ty  of  tha  corralatlon 
maximum,  Thasa  oposs  sact|ons  apa  graphs  of  tha  valua  of 
tha  corralatlon  function  with  ona  of  tha  varlablas  dx  and  dy 
changing  and  tha  otnar  oonstant, 

Flgura  3-0a  shows  tha  corralatlon  function  for  a  window 
containing  tha  crrtar  N|x  0|ymp|oa.  Tha  cruda  30x30  pixel 
windows  ara  data  from  tha  olotures  7F75  and  7F70,  The  first 
graph  Is  the  oorralatlon  funotlon  with  dx  changing  and  dy  = 
8,  Tha  second  graPh  shows  tha  oorralatlon  function  with  dy 
changing  and  dx  *  35,  The  oorralatlon  surface  Is  modeled  by 
fitting  a  second  order  polynomial  In  dx  and  dy  to  Points 
around  the  oorralatlon  peak  <dx  »  35,  dy  s  0),  The  maximum 
oolnt  on  th|s  surface  Is  found  to  be  (dx  a  8.08,  dy  «  35,30) 
attaining  a  correlation  value  of  .75,  using  a 
two-d I  mans  I ona I  Newton's  method, 

Flgura  3«0b  shows  an  area  which  doas  not  ocntaln  any 
particular  features,  The  correlation  function  In  this  area 
Is  s  I  yo  smooth  ».nd  symmatrlc, 

Figures  3-0c  and  3-0d  show  the  behavior  of  the 
oorralatlon  function  whan  low  fraauency  Information  Is 
removed  from  a  picture  (these  areas  are  from  different 
Mariner  6  and  7  olotures),  Figure  3-0c  Is  tha  correlation 
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function  for  an  i\rea  containing  savaral  craters,  Figure 
3-00  shows  the  correlation  funotlon  for  the  seme  area  after 
low  spatial  freauencles  have  been  removed  (subtracting  the 
local  average  Intensity  oomputed  over  e  10x10  area  from  each 
point),  Removing  low  spatial  frequencies  c|ear|y  reduces 
the  width  of  the  oOrrelat|on  peak, 


iii.e  modeling  the  misregistration  AS  a  function  of  position 

IN  THE  IMAGE 

There  are  several  alternative  methods  available  for 
modeling  the  misalignment  functions,  One  method  Is  Mth 
order  Interpolation  (usually  MB2)  In  2  variables  In  |oea| 
regions  of  the  Images,  Another  method  (whloh  was  chosen)  Is 
to  fit  polynomials  |n  2  variables  to  the  entire  set  of 
misalignment  vectors,  and  then  minimize  the  mean  squared 
error  between  the  Polynomials  and  the  empirical  data  points 
(the  method  of  least  squares),  I nte r po | at  I  on  is  good  If  the 
errors  In  the  empirical  data  points  ere  small  (that  Is,  the 
corral  a  cion  search  always  works  well),  and  If  there  are 
enough  data  points  to  adequately  cover  the  entire  Imegt^ 
Polynomial  fits  to  the  entire  Images  were  chosen  because 
they  have  good  smoothing  properties  on  no, sy  data, 

The  misregistration  functions  du(u,v)  and  dv(u»v)  are 
empirically  der|vad  from  n  quintuples  of  the  form 
( uC  I  3#  vc  l  ],  duC  I  3#  dv[  |  ] ,  cU  D  > »  where  < duC 1 3 • dvCI 3 >  Is  the 
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translation  vactor  whjoh  maximizes  the  correlation  oC)] 
between  tha  2  Images  over  a  window  oentsred  at  (uC I D, vC  I  ]} , 
For  a  I  rrD  I  I  o  1 1  y  *  these  functions  ware  chosen  to  be  Nth  order 
polynomials  In  2  variables  of  the  forml 

du(u,v)  ■  sum  AC  I » J 3*u*  I •v* J  (3,10) 

0S  I +JSN 

Tha  traditional  least  squares  approach  Is  to  ohoose  du(u,v) 
euch  that! 

sum  ( du ( uCkU , vCk J ) -duCk3 >*2  ■  m|n  (3,11) 

k 

•  sum  (  sum  AC  I  # J 3 *u C k 3  ♦  I • v C k 3 ♦  J  -  duCk])*2  (3,12) 
k  I ,  J 

This  leaet  sauares  problem  degenerates  to  the  solution  of  an 
(N+l >«(N*2)/2  ord«r  system  of  linear  equations,  The 
polynomial  for  dv  Is  constructed  similarly, 

The  RMS  error  of  the  polynomial  fit  to  the  empirical 
data  Points  Is  a  good  measure  of  whether  the  order  N  of  the 
polynomials  Is  high  enough,  and  If  |n  fact  Polynomials  ere 
the  appropriate  functions  to  f|t  to  the  data,  To  account 
for  the  number  of  degrees  of  freedom  |n  the  polynomial  fit, 
the  following  measure  Is  used) 

(du(uCk],vCk3)-duCk])*2 

sqrt  (  sum  -----------------------  )  (3,13) 

k  Nk  -  ( N*1 ) * ( N  +  2  ) /2 

where  Nk  Is  the  number  of  data  points, 


% 
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Since  the  • mo  I r I c  a |  data  Points  rtprasant  a  sampling  of 
a  9urfaoai  w§  would  like  the  polynomials  to  model  that 
surface  everywhere  as  well  as  sample  points,  A  useful  test 
Is  to  fit  polynomials  to  one  set  of  samo|e  Dolnts»  and  then 
measure  the  RMS  error  between  these  polynomials  and  another 
sampling  of  the  surface, 

The  above  least  squares  bl-varlate  polynomial 
approximations  have  been  used  succassfully  to  model  the 
misregistration  vectors  between  Images,  Tables  3-1  to  3-5 
and  figures  3-1  to  3-7  show  some  actual  misregistration 
vectors  and  the  results  of  fitting  polynomials  of  various 
orders, 

The  tables  contain  the  misalignment  vectors  (UU,DV) 
which  maximize  the  correlation  funotlon  over  a  21x21  pixel 
wlnoow  centered  at  points  <U»V>,  COR  Is  the  maximum  value 
of  the  correlation  function,  The  residual  error  vector 
(DUerr#DVerr>  Is  the  difference  between  the  empirical 
( D  U »  D  V )  vector  and  the  vector  predicted  by  the  polynomial 
model,  ||error||  Is  the  Euclidean  norm  of  the  residual 
error  vector, 

The  figures  graphically  show  the  data  In  the 
oor respond  I ng  tables,  The  square  on  the  left  represents  the 
entire  area  of  the  200x200  pixel  plcturss,  The  vectors  In 
the  square  start  at  points  ( U » V )  and  have  length  and 
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direction  <10*DUerr,10»DVsrr) ,  The  single  digit  et  (U,V) 
represents  the  va'ue  of  the  correlation  maximum  (l,e,  20X 
to  29X  Is  represented  by  2),  The  circle  repreeents  residual 
errors  of  one  standard  deviation.  The  endpoints  of  the 
residual  veotors  ( Due r r , DVer r )  are  plotted  relative  to  the 
oenter  of  the  circle. 

Tables  and  figures  3-1  to  3-3  show  residual  errors 
before  bad  misregistration  vectors  have  been  eliminated  by 
"blunder  removal”  (explained  I  a  te  r  > •  Table  and  figure  3*1 
show  the  residual  errors  from  zero  order  polynomial  fits  to 
14  data  points,  The  predominant  translational  error  Is 
0U*34  and  0V*9  pixels.  The  weighted  RMS  error  is  5,25 
pixels,  where  data  points  are  weighted  aocordlng  to  their 
correlation  value  COR,  Tht  corrected  error  of  equation 
(3,13)  Is  5 , 65  pixels,  To  compare  the  Quality  of  the  fit, 
these  corrected  errors  will  be  used,  Table  and  figure  3-2 
show  the  residual  errors  from  first  order  polynomial  fits  to 
tho  sa.re  data  points,  The  oorrected  error,  wh|ch  Is  4,47 
pixels,  has  not  been  reduoed  much,  Table  and  figure  3-3 
show  the  residual  errors  from  a  second  order  fit,  The 
oorrected  err’jr  Is  4.25  pixels,  which  Is  not  a  significant 
reduct  I  on , 

Tables  and  figures  3-4  and  3-5  show  residual  errors 
4fter  4  "blunders"  have  been  removed,  blunders  are 
m  |  sreg I etr at  I  on  vectors  wnlch  do  not  f|t  any  systematic 
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mor'el,  These  vectors  result  from  cross  correlating  areas 
where  these  |s  some  form  of  systematic  difference  between 
the  pictures  such  as  e|ouds»  or  crater  "shadows”  which  have 
Moved  due  to  changes  In  Illumination.  These  blunders  are 
removed  by  finding  residuals  (DUerr,  DVerr)  which  are  large, 
and  do  not  cluster  with  the  other  res|c!ua|s, 

Table  and  figure  3-4  show  the  residual  errors  after  a 
first  order  fit,  Th#  corrected  error  |s  now  1,03  pixels, 
The  second  order  fit  of  table  and  figure  3-5  has  a  higher 
corrected  error  of  1,18  pixels. 

Figures  3-6  and  3-7  show  the  residual  errors  from  first 
and  second  order  Polynomial  fits  to  a  much  larger  (about  60 
points)  set  of  misalignment  vectors,  after  blunder  removal, 
The  Hrst  order  fit  has  a  corrected  error  of  1,26  pixels, 
which  Is  fairly  consistent  with  figure  3-4,  The  second 
order  fit  has  a  corrected  error  of  1,02  pixels,  which  Is 
somewhat  better  than  the  first  order  fit, 


52 


MliRNPKMsM 


XKaw<  w»  ■  n  Kama  mist.  jwmiwkia>Wp»^i 


5WK 


TABLE  3-1 

POLYNOMIALS  FIT  TO  ORDER  0  BEFORE  BLUNDER  REMOVAL 

DU ( U »  V ) *  34,34 
DV ( U#  V ) *  8,991 


u 

V 

OU 

DV 

COR 

DU  err 

DV  err  ! 

1  terror 

•  60 

-60 

30,16 

10,73 

,68 

-4,17 

1,73 

4,52 

•60 

-  20 

32,01 

20,73 

.87 

-2,33 

11,74 

11.97 

-60 

20 

32,00 

10e00 

,00 

-2,34 

1,01 

2.55 

>60 

60 

32,88 

9,81 

.74 

-1,46 

,82 

1,67 

>20 

-  6  0 

33,92 

4,39 

.25 

-.41 

-4,10 

4.12 

>20 

-20 

31,77 

8,70 

,88 

-2,56 

-.29 

2,58 

>20 

20 

31,07 

7,50 

.82 

-3,27 

-1,49 

3,60 

20 

60 

30,87 

6,66 

.87 

-3,47 

-2,33 

4,18 

20 

-60 

37,17 

7,68 

.77 

2,83 

-1,31 

3,12 

20 

-20 

36,13 

8,69 

.79 

1.79 

-.30 

1,81 

20 

20 

35,65 

6,68 

.71 

1,31 

-2,31 

2,66 

20 

60 

35,18 

6,57 

.85 

.85 

-2,42 

2,56 

60 

-60 

38,00 

7,00 

.00 

3,66 

-1,99 

4.17 

60 

-20 

41,11 

6,00 

,53 

6,77 

-2,99 

7,40 

60 

20 

41,63 

9,23 

,51 

7,29 

,24 

7,29 

60 

60 

39,10 

5,46 

,30 

4,76 

-3,54 

5,93 

TOTAL  WEIGHTED  RMS  ERROR*  5.25 


FIGURE  3-1  RESIDUAL  ERRORS  FROM  TABLE  3-1, 


TABLE  3-2 

POLYNOMIALS  FIT  TO  ORDER  1  BEFORE  BLUNDER  REMOVAL 


DU(U# 

V)* 

34,70  * 

,  76389-1** 

-  ,  8829<«-2*Y 

DV  ( U  1 

V)» 

8,783  - 

,5617»-1*X 

-,1997<a-l*Y 

U 

V 

DU 

DV 

COR 

DU  err 

D V  err  1 

1  e  r  r  0  r 

-60 

-60 

30,16 

10.73 

,68 

-.56 

-2,63 

2.69 

-60 

-20 

32,01 

20,73 

.07 

1,64 

8,18 

8,34 

•  60 

20 

32,00 

10,00 

.00 

1,98 

-1,75 

2,64 

-60 

60 

32,88 

9,81 

.74 

3,21 

-1.15 

3.41 

-20 

•  60 

33,92 

4,89 

.25 

.14 

-6,21 

6,21 

-20 

-20 

31.77 

8,70 

,88 

-1,66 

-1.61 

2.31 

-20 

20 

31,07 

7,50 

,82 

-2,01 

-2,01 

2,84 

-20 

60 

30,87 

6,66 

.87 

-1,86 

-2,05 

2.76 

20 

•  60 

37,17 

7,68 

.77 

.33 

-1.18 

1,22 

20 

-20 

36,13 

8,69 

.79 

-.36 

.63 

.73 

20 

20 

35,65 

6,68 

.71 

-.48 

- ,  58 

.76 

20 

60 

35,18 

6,57 

.85 

-  ,  60 

.11 

.61 

60 

-60 

38,00 

7,00 

.00 

-1,90 

.39 

1.94 

60 

-20 

41,11 

6,00 

.53 

1,56 

.19 

1.58 

60 

20 

41.63 

9,23 

.51 

2,44 

4.21 

4,87 

60 

60 

39.10 

5,46 

.30 

.26 

1,24 

1.27 

TOTAL  WEIGHTED  RMS  ERRORS  3,51 


FIGURE  3-2  RESIDUAL  ERRORS  FROM  TABLE  3-2 


TABLE  3-3 

POLYNOMIALS  FIT  TO  ORDER  2  BEFORE  BLUNDER  REMOVAL 

DU(U»  V>«  33, 35+,085l0#X+. 8858f-3»X2 

-,012*>0»Y-,3867*-3»X«Y<’,6l57»-4»Y2 


DV ( U #  V ) »  8,861- ,05670»X+,9649»-3»X2 


-.76720- 

2*Y+,2504@ 

"3*X*Y 

-,9130<* 

-3*Y2 

u 

V 

DU 

DV 

COR 

DU  ar r 

DV  trr  I 

1  ar  ror 1 

60 

-60 

30,16 

10,73 

.68 

- ,  8  4 

-3,09 

3,20 

60 

-20 

32,01 

20,73 

.87 

,77 

4,90 

4,96 

60 

20 

32,00 

10,00 

,00 

,33 

-4,92 

4,93 

60 

60 

32,88 

9,81 

.74 

,58 

-1,28 

1.41 

20 

-60 

33,92 

4,89 

,25 

1,42 

-2,96 

3,28 

20 

-20 

31.77 

8,70 

,88 

-.35 

-1,57 

1.61 

20 

20 

31,07 

7,50 

.82 

-.86 

-2,26 

2.42 

20 

60 

30.87 

6,66 

.87 

-1,07 

.33 

1.12 

20 

-60 

37,17 

7,68 

.77 

,33 

2,69 

2.71 

20 

-20 

36,13 

8,69 

.79 

,29 

,89 

.94 

20 

20 

35,65 

6,68 

.71 

,63 

-1,02 

1.19 

20 

60 

35,18 

6,57 

,85 

,77 

1.91 

2,06 

60 

-  6  0 

38.00 

7,00 

,00 

-6,01 

1,80 

6,27 

60 

-20 

41.11 

6,00 

.53 

-1.27 

-2,42 

2.73 

60 

20 

41.63 

9,23 

.51 

.67 

.51 

.85 

60 

60 

39,10 

5,46 

.30 

-.62 

-  ,  63 

.89 

TOTAL  WEIGHTED  RMS  ERROR'  2,43 


FIGURE  3-3  RESIDUAL  ERRORS  FROM  TABLE  3-3 
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table  3-4 


POLYNOMIALS  FIT  to  ORDER  1  AFTER  BLUNDER  REMOVAL 


DU  ( U » 

V )  = 

34,04  ♦ 

,  1013*X  - 

,02093*Y 

DV  ( U » 

V  )  = 

7.  "38  - 

,0256*X  - 

,01850»Y 

U 

V 

DU 

DV 

COR  DU 

I  err 

DV  err 

1  terror  1  1 

-60 

-60 

30,16 

10,73 

,68 

.95 

.34 

1,01 

-60 

-20 

32.01 

20,73 

,00 

3,63 

11,08 

11,66 

-60 

20 

32,00 

10,00 

.00 

4,46 

1,10 

4,59 

-60 

60 

32,88 

9,81 

,00 

6,17 

1,64 

6,39 

-20 

-  6  0 

33,92 

4,89 

,00 

.65 

-4,47 

4,51 

-20 

-20 

31.77 

8,70 

,88 

- ,  66 

.08 
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FIGURE  3-4  RESIDUAL  ERRORS  FROM  TABLE  3-4 


TABLE  3-5 


POLYNOMIALS  FIT  TO  ORDER  2  AFTER  BLUNDER  REMOVAL 

DU  (  U  »  V  )  s  33,5l+,1015*X+.2499»-3#X2 

-,01951*Y+. 16B5*-5*X*Y+.1330*-3*Y2 

D  V  (  u  ,  V  )  =  8,003-,02122*X-,1949?-3*X2 

-,01986*Y+.3l0B@-3*X*Y-.76l9?-4*Y2 
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FIGURE  3-5  RESIDUAL  ERRORS  FROM  TABLE  3-5 
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FIGURE  3-6  RESIDUAL  errors  FROM  POLYNOMIALS 

FIT  TO  ORDER  1  AFTER  BLUNDER  REMOVAL 

DU(U.V:=  34,20  ♦  ,1083*X  -,0141*Y 
DV(U.V)*  7,297  -  ,C.T98P-2*X  -,6235<»-2*Y 


t  t 

■ 

cut  1  J-  13  2D.  cut  I  J-  7  ,  n 
n-  2,  STD  DEW  92 


FIGURE  3-7  RESIDUAL  ERRORS  FROM  POLYNOMIALS 

FIT  TO  ORDER  2  AFTER  BLUNDER  REMOVAL 

D  U  (  U  #  V  )  =  33,20+, Ii74«.y*t5789(i»-3*X2 

-,01‘50«Y+,1627(?-4#x*Y+,1373»-3*Y2 

D  V  (  U,  V)  =  7,294-,07  36@-2*X-,8608(i-4«X2 

-,5937?>-2*Y+,6l22(i»-4*X*Y+,5l04(a-4»Y2 


III.F  RENORMAL  I 2AT I  ON  OF  ONE  JMACE  USING  THE  MISREGISTRATION 
MOOEL 


Qnoe  tha  geometric  ml sreg I atrat | on  between  two 
normal  I za d  Imagas  A  and  5  has  been  detected  and  modeled,  a 
naw  Imaga  B '  can  ba  ganaratad  using  this  modal  such  that  B' 
Is  In  gaomatrlo  allgnmant  with  Imaga  A  according  to  aauatlon 
3,3,  In  practice,  this  raaulras  tha  subat I  tut  I  or  of 
u+du(u,v)  for  u  and  v+dv(u,v)  for  v  In  equations  (2,12)  to 
(2,17), 

Figures  3-9a  to  3-l0b  show  the  results  of  the 
registration  techniques  prase n ted  In  this  chapters  Figures 
3-9a  and  3-9b  are  the  same  as  figures  2-6a  and  2-6b,  Figure 
3-10a  Is  the  same  as  figure  3-9a,  Figure  3-10b  shows  the 
result  of  renormalizing  7F7B  to  align  with  7F7>  using  the 
saoond  order  registration  polynomials  from  figure  3-7, 


gure  3- 
78  A  I  I  g 
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CHAPTER  IV 

PHOTOMETRIC  NORMALIZATION  BY  "DEAD  RECKONING" 

Th«  photometr I c  normalization  problem  |s  to  determine 
two  functions  Rl(u*v>  and  R2(u,v)  which  define  the 
reflectivity  of  the  scene  at  point  <u,v)  |n  the  respective 
Images  (and  therefore  at  point  T(u,v)  |n  the  scene),  The 
reflectivity  function  can  be  determined  by  "dead  'ackonlng" 
(from  accurate  calibrations  and  models)  If  the  response  of 
the  Vldlcon  and  the  reflectivity  function  of  the  scene  as  a 
function  of  the  location  |n  the  scene  and  the  Illumination, 
view*  and  phase  angles  are  precisely  known,  Images  can  be 
ohotometr | ca I  I y  registered  with  resoect  to  some  type  of 
errors  |n  the  photometric  model,  As  for  the  geometric 
model,  the  combination  of  the  dead  reckoning  and 
misregistration  models  Is  us^d  to  define  the  reflectivity 
f unct Ion  R( u, v  )  , 

Given  the  geometric  and  photometric  normalization 
functions  (R1*T1)  and  (R2,T2)  for  a  pair  of  Images,  the 
cloture  comparison  problem  becomes  one  of  comparing  the 
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reflectivities  of  cor  respond  I ng  points  In  the  Images*  l.e*» 
for  e||  point,  palps  ( ul  *  vi  )  *  ( u2  *  v2 )  such  thet 
T1 ( ul , vl )  =  T2( u2* v2 ) ,  compape  Rl(ul*vl)  w , th  R2(u2,v2), 

IV, A  PHOTOMETRIC  MODEL  FOR  THE  SCENE 

The  scattering  of  light  by  a  surface  Is  a  very  comp|e> 
function  of  the  spectral  composition  and  polarization  of  the 
Incident  light,  the  composition  of  surface  material,  end  the 
various  angles  between  the  Incident  light  and  the  surface 
(angle  I)  the  emergent  light  and  the  surface  (angle  e)  and 
between  the  Incldant  light  and  the  emergent  light  (phase 
angle),  For  some  surfaces,  w|th  known  light  sources,  the 
reflectance  function  !s  fairly  well  understood. 

The  reflectance  function  of  the  moon  can  be  modeled  by 
the  Minnaert  |aiw  (M|nnaert  C19613)! 

Eout/E|n  *  cos(t)  *  A  •  (cost t ) »eo3( I ) ) 'k  (4,1) 

where  A  is  the  albedo  of  the  surface,  E|n  end  Eout  ar*  the 
light  energy  Incident  to  and  emergent  from  the  surface,  and 
k  Is  a  parameter  particular  to  the  surface  and  the  phase 
ang I  a , 

Young  has  found  (Young  C 1 9 7 1 3 )  that  regions  of  Mats 
covered  by  the  Marlnur  6  end  7  pictures  obey  the  Minnaert 
law  fairly  we  I  | J  he  found  values  for  k  ranging  from  ,46  to 
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.71,  and  values  for  A  ranging  from  ,071  to  ,146, 


Pollack  has  shown  <  Po |  I  aok  C1969J)  a  phase  angle 
dependence  for  the  value  of  k,  Although  actual  data  exists 
for  only  a  limited  number  of  different  phase  angles,  Pollack 
suggests  that  It  Is  reasonable  to  Interpolate  the  following 
tables 


TABLE  4-1 


obias 

ICBlt 

Is 

0 

deg, 

,5 

50 

,7 

180 

1,0 

I V , B  INTENSITY  MODEL  between  data  points 


To  compare  two  Images  (Fl» T1 )  and  (F2,T2)  which  were 
taken  from  from  different  camera  positions  and  orientations 
It  Is  necessary  to  compare  plue|s  |n  Image  l  wltn  those 
Pixels  In  Image  2  which  correspond  |n  the  scene,  For 
discrete  Images,  exact  pixel  cor respondenoe  does  not  exist 
since  the  projections  of  the  pixels  from  the  Images  onto  the 
planet  w I  |  I  not  match  In  location,  size  and  shape,  and  since 
the  orientation  of  the  Image  coordinate  systems  (scan 
direction)  may  be  different, 

Consequently,,  |n  order  to  compare  such  Images  It  Is 
necessary  to  model  the  Intensity  function  between  given 
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pixels  (Rosenfeld  C 1 9 6 9 3 ,  pp20-21), 

2ER0  OROER  MODEL:  The  simplest  model  for  the  Intensity 
function  F  at  non-grid  points  Is  tha  0th  order  model 
F0 { u • v  >  I 

F  0  (  u  1  V  )  «  F(|,.J)  {A, 2) 

where  (l,j)  Is  the  closest  sample  point  to  (u,v),  If  one 
assumes  that  F  Is  defined  with  (l,J)  spaced  on  Integer 
valuesi  then  the  0th  order  model  F0  Is  equivalent  to: 

F0<u,v>  •  Ft Cu*.5J»Cv*,5 3)  (4,3) 

where  Cx]  means  the  greatest  Integer  which  does  not  exceed 

x. 


FIRST  ORDER  MODEL:  The  l»t  order  model  Fl(u,v)  |s 
bilinear  Interpolation  which  computes  a  weighted  average  of 
F  at  the  four  sample  points  which  are  nearest  to  (u,v), 

Fl(u.v)  s  wl»rU»J>  *  w2*F  (  I  +1  *  j  > 

♦w3*F ( | , j+1 )  ♦  w4»F( |*1, j+1)  (4,4) 

where 


l=Cu]#J«Cv] 

<4, 5a) 

psu-Cu3#  qsv*Cv3 

(4,5b) 

wls ( 1-p ) • ( 1 -q  > 

(4,5c) 

w2s ( 1-p ) #Q 

(4,3d) 

w3=o* ( 1-q ) 

(  4 , 5  e  ) 

w4su*q 

(  4 , 5  f  ) 
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Not#  that  0SwCIJSl  and  wl+w2*w3+w4»l , 


HIGHER  ORDER  POLYNOMIAL  MOOELSl  If  higher  order 
surfaces  are  used  for  Interpolation,  Instead  of  the  bilinear 
model,  one  can  require  either  that  eaoh  pleoe  of  surface 
paes  exactly  through  a  large  number  of  sample-point 
Intensities,  or  alternatively,  one  can  reoulre  that  the 
Places  agree,  not  on|y  In  Intensity,  but  also  In  the  values 
of  the  various  derivatives,  along  their  curves  of 
Intersection,  so  that  the  resulting  Interpolated  au'faoe  Is 
not  only  continuous  but  a|eo  smooth, 

SAMPLING  THEORY  MODEL!  Prom  a  sampling  tneory  point  of 
view,  a  discrete  Image  can  be  thought  of  as  a  bandwidth 
limited  function  of  two  variables,  where  the  point  spread 
function  W  determines  the  maximum  frequency  contained  In  the 
discrete  Image,  Given  th | s  model,  the  function  F  can  be 
exactly  r eoonst r Uc ted  from  the  sample  points  using  a 
2-d I  mens  I ona  I  generalization  of  the  Shannon  sampling  theorem 
(Shannon  C1949],  Prosser  C 1966  3  pp,  574-584),  However, 
this  mathemat I oa | I y  sound  model  for  F  Is  exoeed|ng|y 
expensive  oomputat | ona II y ,  anq  of  Questionable  practical 
Improvement  over  simpler  models  for  the  Intended 
app I  I  cat  Ion, 

If  It  were  necessary  to  measure  the  geometric  positions 
of  edges  of  featuree  suoh  as  oraters  w|th  errors  of 
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fractions  of  a  pixel#  than  a  higher  order  modal  would  ba 
nacasstry,  For  the  Mariner  #71  variable  features  problem, 
photometric  accuracy  |s  more  Important  than  such  geometric 
atcuraoy,  and  conseauent I y »  the  first  order  model  (bilinear 
Interpolation)  was  selected  because  of  toth  Its 
computational  simplicity  and  Its  smoothing  properties, 
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CHAPTER  V 

accurate  photometric  recistratjon  of  normalized  images 

V , A  formal  definition  of  PHOTOMETRIC  misregistration 

Assume  that  w8  have  two  photometrically  normalized 
Images  w|th  ref|ectance  (albedo)  functions  Rl(u#v>  and 
R2(u#v>,  which  are  e|so  geometrically  normalised  and 
registered  as  described  In  Chapters  II  and  HI,  Assume  also 
that  the  »ctu»l  scene  reflectivity  Is  the  same  for  the  two 
Images,  The  Images  are  said  to  be  "photometrically 
mlsreglstered  relatively"  at  point  (u,v)  |f» 

Rl(UiV)^R2(u»v)  (5,1) 

If  R(u,v)  Is  the  true  scene  ref|ect|v|ty  then  an  Image  R1  Is 
said  to  be  "photometrically  mlsreglstered  absolutely"  at 
DOlnt  <UiV>  If! 

Rl(u»v)*R<u»v>  (5,2) 

The  following  sections  present  a  model  for  photometric 
misregistration  an<j  derive  functions  which  describe  this 
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m I sreg I strat I  on, 

V.B  MODEL  FOR  MISREGISTRATION 

To  photometrically  register  Images  more  accurately  than 
Is  possible  with  t  '•  "dead  reckoning"  photometric 
normalization  requires  that  wa  have  a  model  for  photometric 
misregistration,  There  are  many  potential  sources  of 
photometric  errors  In  Images  which  hay®  been  "normalized"  by 
Chapter  I v (  Some  of  these  are* 

1 )  Camera  errors 

a)  Errors  In  light  transfer  function 

b)  Non-un|form  sensitivity  across  v|<j|con  surfaoe 

c)  Residual  Images 

d)  Coherent  noise 

e)  Random  noise 

2)  Photometric  model  errors 

a)  Reflectance  function  variations  with  position  on  planet 

b)  Inappropriate  values  of  k  In  equation  (4,1) 

A  general  model  for  photometric  misregistration  would 
be  to  assume  that  there  Is  a  function  P  which  describes  for 
each  picture  the  combined  photometric  characteristics  of  the 
surface  of  the  p|anet»  the  response  of  the  vldlcon*  and  the 
photometric  model  of  Chapter  IV,  If  the  photometric  errors 
are  spatially  uniform*  that  |s»  the  errors  do  not  depend  on 
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their  looatlon  on  the  surface  of  either  the  vldlcon  or  the 
olanet,  then  the  ohotometrlc  registration  funotlon  P  Is  a 
function  only  of  the  rafleotlvlty  R(u,v)  of  eaoh  point,  In 
general,  P  Is  a  function  of  a  set  S  of  variables,  such  as 
the  position  on  the  vjdloon  surface,  the  position  on  the 
planet,  and  the  Illumination,  v|ew,  and  phase  angles  as  well 
as  R(u,v),  Than,  for  two  photometrically  normalized  Images 
R1  and  R2,  we  would  have  two  Photometric  registration 
functions  PI  and  P2  suoh  thatl 

Pl(Rl(u, v) »S) )  *  R( u»  v 3  a  P2 ( R2 ( u, v ) , S )  (5,3) 

whjre  R(u,v)  |s  the  true  rafleotlvlty  (albedo)  of  point 
(Ulv),  If  It  is  not  necessary  to  obtain  photometric 
registration  to  true  reflectivities,  then  the  Images  can  be 
photometr I ca I  I y  registered  relative  tv  one  another  using  the 
function  Pl2  whloh  Is  defined! 

P12 ( r , S 3  =  PI' (P2C  r,S> ,S)  <5,«) 
where  PI'  Is  the  Inverse  of  PI,  or 

Rl(U,v)  «  P17 ( R2 ( u, v ) , S )  (5,5) 

The  Mariner  6  and  7  pictures  have  been  processed 
(Rlndflelseh  C1971])  by  the  Jet  Propulsion  Laboratory  (JPD 
to  remove  as  many  of  these  errors  as  possible,  The  h|gh 
ievel  of  coherent  noise  present  |n  the  Mariner  6  and  7 
pictures  required  that  JPl  develop  some  rather  sophisticated 
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selective  filtering  teehnlauesi  There  Is  some  question 
about  the  absolute  photometric  accuracy  of  these  final 
pictures,  and  there  are  still  errors  due  to  the  residual 
Image  processing  In  the  area  of  the  limb  of  the  Planet, 

There  does  not  presently  exist  an  aeeurate  model  for 
the  photometric  errors  present  In  the  Mariner  '69  pictures, 
However,  after  comparing  Images  which  were  processed  by  the 
techniques  of  Chapters  II,  III,  and  IV,  It  was  found  that 
the  predominant  errors  were  due  to  errors  In  absolute  camera 
sensitivity,  Consequent l y,  the  following  simple  photometric 
registration  function  was  used* 

Pl2(r,S)  =  a*r  ♦  b  (5,6) 

where  the  parameter  a  adjusts  for  errors  In  camera 
sensitivity,  and  parameter  b  adjusts  for  any  constant 
offset,  This  model  assumes  that  a  and  h  are  constant  for 
each  point  In  Images  R1  and  R2* 

It  might  be  better  to  allow  a  and  b  to  be  polynomials 
In  the  two  variables  u  and  v,  Such  a  mode|  would  correct 
for  errors  which  are  spatially  dependent,  If  It  Is  possible 
to  empirically  derive  a<u»v)  and  o(u»v)  and  If  we  could 
separate  the  contributions  due  to  the  camera  from  those*  due 
to  the  photometric  modol  of  the  planet,  then  the  empirical 
functions  should  b«  useful  In  r«f|n|ng  our  photometric  model 
for  Ma r  5 , 
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V,c  A  MEASURE  OP  PHOTOMETRIC  MISREGISTRATION 


Obviously*  In  any  discussion  of  arrors  between  sets  of 
points  It  Is  necessary  to  define  some  measure  of  the  "total 
error",  The  conventional  RMS  error  measure  was  chosen 
primarily  because  the  familiar  least  squares  techniques 
minimize  that  measure, 

The  RMS  error  In  photometric  registration  between  two 
pictures  R1  and  R2  Is  then  defined! 

RMS  ( R1  #  R2 )  =  sqrt  {  sum  (<Rl(u,v)  -  R2(u,v)>»2)  /  N)  (5,7) 

u  i  v 

where  N  Is  the  number  of  points  In  the  pictures, 

V,D  CHOOSING  A  MODEL  TO  MINIMISE  PHOTOMETRIC  MISREGISTRATION 

The  purpose  of  this  section  Is  to  empirically  derive 
the  function  P12  which  "best"  registers  pictures  R1  and  R2 
pho tc me t r | ca | | y  In  terms  of  the  RMS  error,  More  precisely, 
we  find  P12(r#S)  such  that! 

RMS(R1,P12(R2, S)  )  =  m | n  (5,8) 

o  r 

sum  l ( R1 ( u , v  )  -  P12 ( R2 ( u*  v ) , S ) ) f2 )  =  m|n  (5,9) 

u*  v 

!-or  the  simple  model  of  eauat|on  (5.6),  minimizing 
equation  (5,9)  Is  equivalent  to  m|n|m|z|ng: 
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E ( Rl *2 )  *  «t2«E(R2*2)  -  2a*E(Rl*R2>  -  2b*E(Rl) 

♦  2ab*t(R2)  *  b»2  =  m|n  <5, If) 

where  L  is  the  e*o®cted  value,  The  solutions  for  a  and  b 
are: 

a  *  cor(Rl,R2J  •  sd(Rl)/sd(R2)  (5, n*) 

b  =  E ( R 1 )  -  a*E ( R2 )  <5,llb) 

where  sd  Is  the  standard  deviation#  and  cor  Is  the  cross 
correlation  defined  by  equation  (3,5), 

If  a  more  complicated  model  Is  used#  where,  a  and  b  are 
functions  of  th#  ( u  #  v )  position  |n  the  Image  formed  by 
linear  combinations  of  functions  Ak(u#v)  and  Bk(u#v),  then 
we  want  to  find  coefficients  atk]  ano  b[kj  to  minimize* 

sum  ((Rl(urv)  -  R2 ( u »  v  )  •  sum ( a C k J * a k ( u » v ) ) 
u »  v  k 

-  sum ( b[ k ] *dk ( U# V ) ) ) ♦ l  c  m|n  (5,12) 
k 

where  Ak  and  Bk  are  Indexed  by  k,  The  values  of  the  a[k] 
and  bCk3  are  found  |n  the  traditional  manner  of  least 
squares.  Intuitively,  first  order  polynomials  In  u  and  v 
appear  to  be  useful  for  the  functions  A  and  B,  Since  this 
technlaue  for  photometric  registration  has  not  yet  been 
useo#  the  proper  order  and  form  for  the  A  and  B  functions  Is 
not  yet  known,  Only  zero  order  models  (equation  (5,6))  have 
ectue I  I y  been  tried, 
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CHAPTER  VI 

oifferencing  ANO  difference  analysis 

The  title  of  this  dissertation  It  "Computer  Comparison 
of  Pictures"!  but  the  previous  four  ohaoters  have  all  dealt 
with  Image  norma  I  I  lat  |  on  and  registration,  and  nothing  has 
really  bean  said  about  Image  comparison,  Of  course,  a||  of 
these  preliminary  operations  were  necessary  so  that  the 
differences  between  the  two  pictures  represent  actual 
changes  In  the  scenas  rather  than  differences  due  to  the 
eonoltlons  of  viewing  and  Illumination, 

This  chapter  describes  teohnioues  for  Image 
dlfferenolng  and  the  analysis  of  these  differences, 

VI, A  POINT  BY  POINT  DIFFERENCES 

The  most  obvious  form  of  differencing  of  two  normalized 
and  registered  pictures  R 1  and  R2  |  s  the  point  by  point 
d I f f ereroe  def I ned  by  5 


Q(u#v)  «  Rl(u»v)  -  R2(u,v) 


(6, la) 


If  the  two  Images  have  not  been  photometrically  registered, 
then  the  following  point  py  Point  difference  also  performs 
the  photometrlo  reg I strat I  oh! 

u<  u, v ;  =  R 1 ( u , V )  -  A(u,v)*R2(u#v)  -  ti(u»v)  (6,1b) 

where  A  and  ti  are  the  photometric  ml sreg I strat I  on 
polynomials  derived  In  Chapter  V,  These  difference  pictures 
show  where  there  are  errors  In  the  data  dye  to  noise  of 
various  forms,  where  there  are  errors  In  the  photometric  or 
geometric  registration  models,  or,  hopefully,  where  there 
are  actual  differences  In  the  reflectivity  of  the  scene, 

Applylnq  the  latter  differencing  technique  to  the 
pictures  In  figures  6-le  and  6-lb  (which  are  the  same  as 
3-108  and  3-10b),  w0  get  the  difference  Pictures  which  are 
shown  In  figures  6-2a  and  6-2b,  figure  6-2a  was  produced  -iy 
subtracting  figure  6-lb  from  6-la  (and  adding  •  constant  so 
that  there  are  n0  negative  numbers),  figure  6-2b  Is  the 
negative  of  6-2a , 

The  most  obvious  feature  of  these  difference  pictures 
Is  the  large  Irregular  bloo  In  the  lower-right  corner,  which 
Is  oarh  In  6-2a  and  bright  In  6-2b,  and  whose  cause  will  be 
explained  later,  There  are  numerous  small  circular  light 
and  dark  spots  which  are  due  to  reseau  marks  on  the 
vldlcons.  (In  future  work,  these  areas  will  always  he 
black),  Reseau  marks  are  deposits  on  the  v|dlcon  surface 
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for  geometric  alignment  of  the  vldlcon  scan,  There  are  also 
light  and  dark  linear  horizontal  streams  wh|cn  are  oriented 
In  the  horizontal  scan  direction  In  the  original  camera 
coordinate  system,  These  streaks  are  probably  due  to 
coherent  no  I  so  |n  the  camera  system  (R|ndf|elsoh  [1971]), 
Slight  geometric  m I s r e g I s t r at | on  Is  Indicated  around  Nix 
Olympics,  the  large  crater  located  at  the  top-center  of  the 
aligned  pictures.  A  high  level  of  visually  uncorre|ated 
noise  Is  also  present.  The  global  dark  to  light  shading 
from  left  to  right  jn  figure  6-2a  can  be  explained  by  errors 
In  the  assumed  ceoendence  of  the  photometric  function  on 
viewing  angle  near  the  limb,  A  linear  model  for  the 
photometric  registration  model  of  equation  (b,l2)  would  have 
reduced  this  error, 

The  on|y  significant  unexplained  difference  between 
figures  6-la  and  6-lb  Is  the  large  blob  which  is  brighter  In 
6-lb  than  In  6-la,  This  blob  has  been  attributed  <leovy 
[1971])  to  a  cloud  consisting  of  water  vapor,  This 
difference  Is  visible  In  figures  6-la  and  6-lb,  but  Is  very 
difficult  to  see  In  the  original  pictures  (figures  2-t> a  and 
2-5b> , 

f  Igures  6-3a  through  6-6b  show  the  results  of 
Irtroduclng  a  third  picture  named  7F77,  which  was  the 
picture  taken  Immediately  before  7F7B  In  the  Mariner  7  fer 
encounter  seouence.  Picture  7F77  was  normal  Ized  to  the  same 
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orthographic  projection  as  was  used  for  7F7o  and  7f7B»  and 
geometrically  aligned  with  7f75,  Tha  figures  show 
differences  between  7F77  and  both  7F7b  and  7F78,  Since  7F77 
was  taken  between  7F75  and  7F78  |n  t|me,  It  Is  reasonable 
that  we  see  an  I  nte  r  wed  I  a'.e  state  of  the.  development  of  the 
cloud, 

The  higher  level  of  noise  In  figures  6«6a  and  6-6b  Is 
due  to  an  Increased  gain  factor  Introduced  by  a  histogram 
"stretching"  program.  This  stretching  Is  used  to  adjust  the 
Intensity  range  of  a  picture  to  maximize  the  contrast  when 
generating  a  photograph,  These  figures  also  show  another 
camera  related  error  across  the  top  which  Is  oriented  In  the 
horizontal  scan  d|r«et!on  of  the  camera,  It  should  Le 
emphasized  that  these  pictures  were  never  geomet r I oa  I  I  y 
registered  with  one  another,  but  both  were  registered  to  the 
910)1  third  picture  (7F75),  There  seems  to  be  little  or  no 
extra  geometric  m I s r eg  I s t r a 1 1  on  due  to  cumulative  errors  as 
might  be  expected, 

VI  ,8  NO  I  St  "'REMOVAL" 

The  high  levels  of  noise  present  In  the  difference 
pictures  suggests  that  some  form  of  no  loo  removel  Is  needed, 
The  obvious  problem  with  noise  removal  Is  that  usually  one 
cannot  distinguish  signal  from  noise,  and  consequent  I y ,  when 
performing  noise  removal  one  a|so  oerforms  some  signal 
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removal  , 

If  the  noise  Is  random,  such  that  the  noise  In  adjacent 
samples  Is  unco r r e I ated *  then  one  way  to  reduce  It  Is  by 
spatial  filtering  or  averaging,  Figures  6-7b»  6-8b»  and 
6-9b  show  the  results  of  replacing  each  point  In  the 
difference  pictures  (6-7a,  6-8a,  and  6-9a)  by  the  local 
average  computed  over  an  llxll^polnt  window,  This  procsss 
effectively  reduces  the  level  of  random  noise,  hut  the 
cloua,  end  the  systematic  errors  |n  onotometry  still  remain, 

Other  typos  of  no|se  removal  have  been  developed 
elsewhere  but  have  not  yet  been  tried  here,  One  such 
technique  Is  "salt  and  pepper”  removal,  which  consists  of 
replacing  points  which  differ  greatly  from  a  surrounding  set 
of  approximately  edulvalent  points,  This  technique  (which 
Is  sometimes  cal  led  "Custerlng"  after  the  well  known 
Arrerlcar  general,  who,  when  surrounded  by  Indians,  was  wiped 
out)  would  remove  most  of  the  ranoom  "snot"  noise  In  the 
difference  pictures  without  introducing  the  blurring  created 
by  spatial  averaging, 

VI, c  contouring  iso-difference  levels 


The 

technique  of 

contouring  Is 

often 

used  in 

ter  restr  1  a 

1  mao-maklng 

to 

emphasize  | 

1 ne  s  of 

constant 

a  1 1 1 tude  , 

Recognizing 

the 

utility  of 

th  |  s 

technlaue, 
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figures  6- 7c ,  6-3c,  and  6- 9c  were  produced  by  contouring  the 
averaged  pictures  6-7b,  6-Pb,  and  6-9p  [1J, 

In  these  figures,  the  sharp  dark  to  white  edges  occur 
at  difference  levels  of  16,  3*?,  and  4J  In  the  co r r e spond | ng 
spatially  averaged  pictures  whose  difference  values  ranbe 
fronr  0  to  63,  ComDutat  I  ona  |  I  y ,  these  pictures  were 
trivially  generated  by  multiplying  each  difference  sample  by 
4,  and  truncatlnq  tne  result  to  6  bits  (0  to  63  range), 

These  contour  pictures  fairly  we|i  outline  the  cloud, 
but  they  also  s^ow  a  lot  of  lines  In  areas  which  seem  to 
contain  no  significant  differences,  In  fact,  one  of  the 
contour  levels  corresponds  to  t  h  o  mean  value  of  the 
dlffarerce  picture,  and  therefore,  on?  would  expect  many 
contours  corresponding  to  small  positive  and  negative 
deviations  from  tne  mean, 

An  Improved  contouring  algorithm  Is  planned  which  will 
select  contour  levels  on  the  cauls  of  tne  histogram  of  the 
difference  picture,  attempting  to  emphas'ze  significant 
differences,  but  not  small  deviations  from  the  mean, 


[13  The  spatially  averaged  pictures  were  used  here  because 
of  the  h|gh  random  noise  levels  In  the  unaveragea  pictures, 
which,  when  contoured,  resulted  primarily  In  garbage, 
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VI, D  DIFFERENCE  "BLOB”  DETECTION 

AnotMr  form  of  analysis  of  difference  pictures  |s 
dlfferenoe  b|op  detection,  whloh  attempts  to  find  the 
outlines  of  areas  (blobs)  of  significant  differences,  The 
significance  of  a  b|ob  Is  determined  by  the  magnitude  of  the 
differences,  and  by  the  length  of  the  edge  surrounding  the 
d  I  f  f  er enoe  blob, 

Figures  6-7d,  6»8d,  and  6-9d  show  the  results  of 
applying  such  a  blob  detector  to  the  cor  respond  I ng  averaged 
pictures  6-7b,  6-8b,  and  6«9b, 

A  further  refinement  of  the  blob  deteotor  (Hannah 
C19713)  attempts  to  determine  whloh  of  the  undifferenced 
olotures  oontalns  the  difference  blob,  This  determination 
Is  made  by  looking  |n  the  und  |  f f e renced  pictures  fop  edges 
In  the  vicinity  of  the  edges  of  the  d[ffepenoe  blob,  An 

edge  Is  defined  statistically  In  both  olotures  |n  terms  of 
the  means  and  variances  both  Inside  and  outside  of  the 
difference  blob,  If  the  vaplanoe  of  one  picture  )s 
significantly  larger  than  that  of  the  other  (In  the  vicinity 
of  the  blob)*  then  the  blob  Is  attributed  to  the  picture 
with  the  larger  variance,  Otherwise,  |f  the  absolute 
difference  In  the  means  Inside  and  outside  of  the  blob  Is 
larger  In  one  picture  than  In  the  other,  then  the  blob  Is 
attributed  to  the  picture  with  the  larger  difference,  This 
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technique  Is  still  being  developed  and  will  be  valuable 
d I f f erence  ana  I  ys | s  , 

Other  difference  analysis  techniques  are  planned* 


not  yet  Implemented, 


CHAPTER  VII 

COMPUTER  IMPLEMENTATION 

Th#  techniques  described  In  this  dissertation  were 
developed  for  use  on  e  digital  computer,  Analog 
I  mo  I ementat | on  of  these  techniques  Is  very  limited  because 
of  the  difficulty  In  performing  the  necessary  photometric 
and  geometric  normalizations  and  registrations  using  analog 
devices, 

Except  where  otherwise  noted,  all  of  these  techniques 
have  been  Implemented  by  the  author  using  the  POP-10 
computer  at  the  Stanford  Artificial  Intelligence  Project, 
This  computer  Is  attached  to  128K  c i k  s  1024)  36-blt  words 
of  core  memory,  and  executes  Instructions  at  the  rate  of 
approximately  one  every  3  microseconds,  The  PDP-10  Is 
"time-shared"  among  typically  20  users  at  a  time,  making  |t 
possible  for  many  users  to  duveloD  and  debug  programs 
Interactively  with  very  short  "tur n-ar ound"  times, 
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Most  of  the  programs  have  beep  written  In  SAIL 
(Swlnahart  C1970]),  a  local  dja|ect  of  ALGQL-60,  and  many  of 
the  time  consuming  Inner  loops  have  Been  written  |n  the 
assembly  language  embedded  In  SAIL, 

VII, A  BLOCK  DESCRIPTION  of  the  programs 

The  programs  are  divided  both  logically  and  Physically 
Into  blocks  which  operate  Independently,  Most  of  the  major 
blocks  of  figure  7-1  correspond  to  the  techniques  described 
I  n  Chapters  1 1  through  vl , 

Images  are  represented  as  packed  arrays  of  light  (or 
albedo)  values,  and  art  accessed  using  PUP-10  byte 
operators,  As  many  samples  as  Possible  are  packed  |n  a 
word,  Usually,  either  6  or  V  b|t  samples  are  used,  giving 
either  6  or  4  samples  per  36 - b 1 t  word,  Thus,  an  Image 
consisting  of  200  Samoles  Der  1 | ne ,  200  lines,  and  6  samples 
per  wore  would  require  6667  words  of  storage,  Procedures 
are  available  to  transfer  pictures  between  core  and  disk, 
and  to  display  Images  on  either  of  two  video  synthesizers 
<  sect  lop  v I  I  ,  B  )  , 

In  the  block  diagrams,  rectangular  blocks  Indicate 
functional  operations,  and  oval  b|oc*s  Indicate  data,  The 
description  of  each  b|cck  follows) 
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F|gur«  7-1  Block  Diagram  of  Syatam 
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Vi  i  ,A,1  NORMAL  1 ZE 

NORMALISE  Is  the  name  of  the  block  which  I mD I om®nts  the 
norrra  I  I  zat  I  on  techniques  of  Chapters  II  and  IV,  The 
Internal  structure  of  NORMALIZE  Is  shown  |n  some  detail  In 
figure  7-2,  The  major  functions  of  NORMALIZE  are  geometric 
and  photometric  norma | I zat I  on  based  on  models  for  camera  and 
scene  geometry  and  photometry, 

USER  INTERACTION:  The  block  labelled  "user  Interaction" 
Indicates  control  of  the  following  parameters  for  the 
orthographic  projections 

1)  Direction  (center  latitude  and  longitude)  of  the 

project  I  on, 

2)  Pixel  size  In  the  projected  Image  (kilometers  par 

p  I  x  o  I  )  , 

3)  Horizontal  and  vertical  size  (In  pixels)  of  the 

projected  Image, 

4)  Rotation  of  the  projected  Image, 

5)  Position  of  the  center  of  the  projected  Image  on  the 

p I anet , 

The  combination  of  the  models  and  the  user  Interaction 
generates  a  set  of  projection  parameters  which  control  the 
actual  normalization  procedure  labelled  " PROJECTOR", 

PROJECTION  ANALYZER:  This  block  analyzes  the  geometric 
calibration  data  for  the  spacecraft  (l.e.  spacecraft 
position  and  orientation  at  the  time  the  picture  was  taken) 
to  determine  the  area  on  the  surface  whloh  Is  covered  by  the 
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Figure  7-2  Block  Diagram  of  NORMALIZE 


Dlcture,  This  analysis  |s  usually  done  for  pairs  of 
pictures  to  determine  the  surface  area  which  |s  Included  fn 
both  pictures,  a  graphical  display  of  the  spacecraft  and 
planet  geometry  Is  presented  to  the  Interactive  user  to 
enable  him  to  specify  the  desired  projection, 

A v G k I N i  S|nce  the  Mar|ner  '69  pictures  are  quite  large  (180K 
words)  It  Is  necessary  to  operate  on  windows  of  entire 
Pictures,  spatial  averages  of  Pictures,  or  combinations  of 
windows  and  averages,  AVGWJN  computes  an  Nx  by  Ny  soat|a| 
average  of  an  arbitrary  rectangular  window  of  a  picture, 
where  \x  and  Ny  are  the  numbers  of  Pixels  averages  In  the 
vertical  and  horizontal  directions,  If  pixels  of  the 
reaulred  projected  Image  correspond  In  s|:.e  to  a  very  large 
number  of  pixels  In  the  raw  Image,  then  sp.t  a|  averaging  Is 
mathematically  necessary  because  of  the  sampling  theorem, 
and  computationally  useful  to  reduce  the  memory  size 
required  for  the  raw  Image, 

PROJECTOR  t  This  b;ock  generates  a  geometrically  and 
photometrically  normalized  picture  from  a  raw  or  averaged 
raw  picture  (which  Is  hopefully  correct  In  light  values  and 
camera  geometry)  using  the  normalization  parameters  and 
reflectance  parameters,  For  each  point  In  the  projected 
Image,  PROJECTOR  calculates  Its  position  In  the  raw  Image, 
Since  this  position  Is  usually  not  exactly  at  a  sample 
point,  a  bilinear  Interpolation  of  the  light  values  of  the 
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four  closest  saints  Is  calculated,  PROJECTOR  also 
calculates  the  sun#  view#  and  phae#  angles#  which  together 
with  th#  light  value#  ylald  th«  a|bado  using  eouatlon  (4,1), 

REGISTRATION  POLYNOMIALS!  NORMALIZE  perform*  the 
renormalization  orooedur*  described  In  seotlon  1 1 1  v  F  using  a 
pair  of  polynomials  produoed  by  the  geometric 
m  1  sreg I strat I  on  mode  I , 

VII  ,A,2  MATCH 

This  b|ook  Implements  th#  techrilaues  f0r  determining 
geometrlo  misregistration  described  In  Chapter  III,  The 
Internal  structure  of  this  block  |n  shown  In  more  detail  |n 
figure  7-3,  Selected  nxn  windows  of  two  normalized  Images 
are  oross  correlated  according  to  equation  (3,3),  A  searoh 
Is  made  for  the  veotor  < duC 1 3 # dvC 1 3 )  which  maximizes  the 
value  of  the  correlation  funotlon  for  Integer  values  of  du 
and  dv,  The  correlation  surface  |s  modeled  by  polynomials 
In  two  variables  In  the  vicinity  of  the  maximum  using  a  2-D 
least  SQuaree  program,  The  maximum  value  of  this  surface  Is 
found  by  computing  partial  derivatives  finding  the  zero  of 
the  Euclidean  norm  of  the  partial  derivatives  of  the 
surface,  using  a  tw0-d I  mens  I  one |  genera  I  I zat I  on  of  Newton's 
Method  ( sect  Ion  1 1 1 , 0 )  , 
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Figure  7-3  Block  Description  of  match 


V  J  I  i  A , 4  PHOTOMETRIC  REGISTRATION 

This  block  Implements  the  photometric  registration 
technlaues  described  In  Chapter  V,  It  oomputes  the 
registration  parameters  which  minimize  the  photometric 
misregistration  between  the  geometrically  aligned  Images, 

V  J  I  i  A  ,  5  0 I FFERENCER 

The  OIFFERENCER  Implements  the  pixel  -by-p I xe I 
differencing  described  In  seotlon  VI, A,  The  dlfferencer 
uses  the  photometric  registration  model  generated  by  the 
technlaues  of  Chapter  V  to  compute  the  difference  picture 
defined  by  equation  <6,lb), 

V i I , A , 6  DIFFERENCE  ANALYSIS 

This  block  performs  the  difference  analysis  technlaues 
described  In  Chapter  VJ,  and  Is  shown  |n  more  detail  fn 
figure  7-4 , 

9b 


V I  I , B  IMAGE  DISPLAY 


Images  which  are  generated  by  any  of  the  Image 
generating  technlaues  of  this  dissertation  can  be  displayed 
by  either  of  two  Image  dlsolay  devices,  Both  of  these 
hardware  devices  are  controlled  by  programs  which  Input 
pictures  from  the  disk  file  system*  and  output  the  pictures 
on  a  selected  area  of  the  Image  generation  surfaoe,  which  In 
each  case  consists  of  a  cathode  ray  tube, 

VII.8,1  HIGH  RESOLUTION  VIDEO  SYNTHESISER 

The  high  resolution  video  syntheslzor  consists  of  a 
collection  of  digital  and  analog  circuitry  to  control  the 
position  and  Intensity  of  the  beam  of  a  Tektronix  model  611 
storage  oscilloscope  which  Is  used  In  non-storage  mode, 
This  synthesizer  generates  grey  level  Pictures  one  line  at  a 
time  as  follows? 

a,  The  611  oscilloscope  beam  Is  positioned  at  the  <x,y) 
coordinates  of  first  point  on  the  line. 

B,  The  611  oscilloscope  beam  Is  turned  on  for  a  duration 

proportional  to  the  Intensity  value  for  this  point, 

C,  The  beam  Is  deflected  to  the  right  one  sample  position, 

and  control  Is  returned  to  step  B  If  there  are  still 
points  to  display  on  th|s  ||ne, 

D,  When  the  line  Is  finished  control  returns  to  step  A, 
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F|gure  7-4  Difference  Analysis 
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The  hardware  for  the  high  resolution  video  synthesizer 
Is  theoretically  capable  of  4096-  by  4096-polnt  spatial 
resolution  with  512  levels  of  flrey,  The  system  Is  currently 
limited  by  the  capabilities  of  the  611  oscilloscope,  which 
limits  the  effective  resolution  to  approximately  700  by  700 
points,  with  approximately  64  resolvable  levels  of  grey, 

The  amount  of  time  required  to  display  a  picture 
depends  on  both  the  size  of  the  Picture  and  Its  content, 
Bright  points  require  that  the  beam  be  turned  on  longer  than 
dark  points,  T y d I c a | I y i  a  picture  requires  25  microseconds 
per  point,  This  means  that  a  200  by  200  picture  would  be 
generated  In  one  second,  In  order  to  view  pictures  which 
are  generated  so  slowly,  a  photograph  must  be  taken,  a 
Polaroid  camera  Is  used  for  this  purpose, 

VII, B, 2  REAL  TIME  VIDEO  SYNTHESI2ER 

The  real-time  video  synthesizer  consists  of  a  different 
collection  of  digital  and  analog  circuitry  which  generates  a 
television  picture*  using  as  Input  8  channels  from  a  10  MHz 
digital  disk  (Data  Disk),  These  8  digital  channels  contain 
the  bits  of  the  b|nary  coded  sample  values,  which,  when  used 
as  Inputs  to  a  digital  to  analog  convertor,  generate  one  of 
256  analog  vo  I  tages , 

The  picture  consists  of  480  scan  lines  of  S>  1 2  Points 


D«r  line,  with  256  levels  of  grey  (approximately  128  usable 
levels),  This  picture  can  be  either  viewed  directly  or 
ohotogrepned , 

VII, C  EXECUTION  TIMES 

Th|e  section  describes  the  CPU  time  required  for  some 
of  the  Important  and  time-consuming  parts  of  the  system  (CPU 
time  refers  to  actual  orocessor  time,  rather  than  elapsed 
time),  These  times  are  averages  over  several  passes  through 
the  system,  and  ar*  only  approximate, 


The 

CPU  times 

to  generate 

the 

200x200  pixel 

Pictures 

shown  In 

this  d | sser tat | on  are 

as  foil ows : 
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45 
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16  correlation 

windows 

220 

sec  i 
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81  correlation 

w 1 ndows 

POLYFIT 

4 

sec 
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1st  order 

There  are  the  on|y  programs  whose  exeoutlon  times  have 
been  measured,  N0ne  of  the  other  Programs  require  times 
comparable  to  e|ther  PROJECTOR  or  MATCH, 
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chapter  VIII 

CONCLUSION 


This  final  Chapter  describes  what  has  been  accomplished 
towards  solving  tne  Mariner  '71  Variable  features  Problem, 
how  this  contributes  to  the  general  science  of  Image 
processing,  and  o|ans  for  future  refinements  and 
applications  of  these  techniques, 

VI  I  I .A  accomplishments 

The  primary  goal  of  this  research  was  tne  development 
of  Image  processing  techniques  to  compare  Pictures  of  Mars 
which  were  taken  from  space  under  different  conditions  of 
viewing  and  |Mum|nat|on«  The  solution  was  to  normalize  the 
pictures  to  the  same  projection  using  geometric  and 
photometric  models  for  the  camera  and  the  scene,  Geometric 
and  photometric  registration  techniques  were  developed  to 
remove  the  effects  of  errors  |n  these  models. 
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A f t • r  pictures  have  been  normalized  and  registered, 
they  are  analyzed  for  differences,  host  of  the  differences 
which  have  been  found  between  the  Mariner  6  and  7  clotures 
are  due  to  noise  |n  the  cameras.  Some  noise  reduction  and 
difference  analysis  techniques  have  been  develoced  to  aid 
the  analysis  of  the  difference  pictures, 

The  examples  presented  In  this  pacer  show  the  detection 
of  a  real  difference,  due  to  a  o|oud  on  Mars,  The  system 
has  been  tested  with  other  Mariner  6  and  7  pictures,  which 
have  not  displayed  Such  dramatic  and  recognizable 
d  I  f f erences, 

The  Jet  Propulsion  Laboratory  devised  a  test  of  the 
capabilities  of  People  and  oomcuters  to  detect  variable 
features,  JPL  Introduced  artificial  differences  of  5%  In 
reflectance  Into  a  Mariner  6  picture,  This  modified  picture 
was  then  geometrically  distorted  to  what  JPL  olalmed  was  a 
different  perspective  view  of  Mars,  The  test  was  to  s##  If 
oeople  (or  computers)  could  determine  the  differences 
between  the  original  picture  and  the  modified,  distorted 
oloture, 

The  result  was  that  oeople  were  unable  to  see  the 
differences,  but  the  system  described  In  th|s  dissertation 
was  able  find  most  of  the  differences,  The  only  differences 
that  were  difficult  to  detect  oecured  because  the  JPL 
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distorted  picture  was  not  a  true  perspective  view  of  the 
planet#  and#  after  geometric  normalization  and  registration, 
some  high  order  resjdua!  distortions  remained, 

No  systematic  difference  analysis  of  the  Mariner  6  and 
7  pictures  has  been  attempted  yet,  However,  this  analysis 
Is  planned,  to  test  the  Mars  global  model  which  Is  described 
I  n  sect  Ion  VI  I  I ,C , 

VI  I  I ,d  CONTRIBUTIONS 

The  major  contributions  of  this  research  to  the  science 
of  picture  processing  are  summarized  as  f  o  I  lows* 

1)  Accurate  geometric  registration  based  on  cross 

correlations  Although  cross  correlation  has  been  used 
In  character  recognition  and  scene  congruence 
(Flschler  [1971]),  apparently  no  one  has  derived  a 
model  for  the  m | s r e g I s t r a 1 1  on  between  two  pictures  and 
used  this  m0del  to  produce  two  pictures  which  are  |n 
geometr I c  all  gnment , 

2)  Photometric  normalization  using  the  M|nnaert  scattering 

model , 

3)  Development  of  a  system  which  combines  a  large  variety  of 

Image  processing  techniques:  Some  of  the  techniaues 
described  In  this  paper  have  been  used  elsewhere,  but 
have  never  been  combined  In  a  coherent  system  which 
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actually  works  with  real  (rathar  than  synthet(c) 
o I oturas* 

4)  Tha  development  of  an  Image  processing  system  for 
Interaction  with  a  planetary  exploration  mission: 
Although  the  system  has  not  actually  bean  tested  |n 
Its  Intended  application*  It  should  satisfy  most  of 
the  needs  of  variable  features  detection,  The  ability 
to  rapidly  detect  changes  on  Mars  provides  the  ability 
to  I  n  o  :*  e  a  s  e  the  scientific  return  from  the  mission, 

VI  I  I ,C  FUTURE  FLANS 

Further  research  Is  planned  to  refine  the  techniques  In 
this  paper#  to  apply  these  teohnlaues  to  new  Image 
processing  problems,  and  to  develop  new  capabilities  fop  the 
Mar  I  ner  '71  Mission, 

REFINEMENTS! 

Several  Improvements  |n  the  correlation  maximization 
search  technique  are  Planned  which  will  use  Information 
about  the  s | gna  | - to-no I se  ratios  and  autocorrelation 
functions  of  the  p  ctures  to  adjust  various  search 
parameters. 

The  higher  order  photometric  misregistration  model  of 
equation  (5,12)  will  be  Implemented,  This  model  should  help 
to  refine  the  parameter  <k  |n  equation  (4,1))  of  the 


M | nnaert  f unct I  on, 

Some  new  picture  difference  operators  are  planned  which 
would  minimize  the  effects  of  one  pixel  ml  area  |  strat  I  one, 
This  technldue  finds  the  dlffereno*  between  a  point  In 
picture  A  and  a  point  |n  picture  ti  which  Is  closest  In 
Intensity  value  and  a  neighbor,  An  attempt  w| | |  be  made  to 
Introduce  symmetry  into  this  operator, 

Tests  of  ’’significance"  of  differences  need  to  be 
developed,  Presumably,  these  tests  would  be  based  on  a 
noise  model  for  the  camerai  and  some  knowledge  of  the  types 
of  differences  to  expect. 

OTHER  APPLICATIONS* 

Researoh  Is  planned  which  would  use  the  cross 
correlation  registration  technique  to  match  areas  of  two 
parallax  views  of  a  scene,  Using  a  set  of  match  points,  a 
deoth  model  for  the  scene  will  be  derived, 

NEW  CAPABILITIES  FOR  THE  MARINER  1971  MISSION; 

Additional  capabilities  are  reau|red  for  the  Mariner 
'71  Mission,  To  detect  variations  on  Mars  whloh  occur  over 
long  periods  of  time,  It  Is  necessary  to  maintain  a 
"library"  of  previous  pictures,  New  pictures  would  be 
oomparec  with  previous  pictures  of  the  same  area  on  the 
sur  face , 
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lach  cloture  contains  approximately  5  million  bits  of 
data,  which  can  b«  compressed  by  3*1  using  Huffman  coded 
pixel  differences,  At  |&ast  1000  Pictures  w|th  controlled 
sun  and  view  angles  are  to  be  processed  for  differences, 
With  the  data  compression!  this  amounts  to  1,7  b I  |  I  Ion  bits, 
which  1 8  about  three  times  the  storage  capacity  of  our 
present  disk  file  system.  This  quantity  of  data  obviously 
will  reaulre  a  carefully  designed  data  management  and 
retrieval  system, 

A  data  structure  Is  planned  wh|eh  will  be  organized  by 
area  on  the  surface  and  by  urlolnal  Picture, 

The  surface  of  Mars  will  be  divided  Into  convenient 
areas,  probably  10  degrees  latitude  by  10  degrees  longitude, 
for  each  area  a  list  will  be  maintained  (on  the  disk  f||e) 
of  the  following  Information* 

1)  A  list  of  all  pictures  which  oover  this  area, 

2)  A  reference  Image  for  this  area,  This  Image  represents 

this  area  to  the  best  of  our  knowledge.  The  reference 
Image  Is  probably  created  by  averaging  pictures  which 
do  not  demonstrate  significant  differences  In  this 
area,  An  Image  which  Shows  the  Pixel  by  pixel 
variances  of  these  p’etures  should  probably  be 
associated  with  the  reference  Image, 

3)  Control  points,  Control  points  are  features,  such  as 
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craters*  whose  positions  are  accurately  Known,  These 
oolnts  are  useful  for  registering  Images  to  fixed 
surface  coordinates,  For  each  control  point,  the  data 
structure  should  Include  Its  planet  coordinates,  and  a 
small  Image  of  the  control  Dolnt  for  use  by  the 
correlation  Procedure, 

4)  A  list  of  any  significant  differences  seen  In  this  area, 
For  each  such  difference  there  might  be  a  difference 
Image,  a  contour  map,  a  b|ob  outline,  or  some  other 
representation  of  the  difference.  The  actual 

difference  Information  need  not  be  currently  stored  on 
the  disk  file  system,  cut  Information  must  be  present 
to  retrieve  It  from  magnetic  tape, 

For  each  picture  the  following  list  w|||  be  maintained* 

D  Picture  Identification  (e,g,  7F75), 

2)  Calibration  data,  l.e,  spacecraft  position  and 

orientation,  and  sun  angle, 

3)  List  of  surface  areas  covered  by  this  picture, 

4)  Location  In  the  magnetic  tape  library  of: 

a)  Original  version  of  the  picture, 

b)  Locally  processed  version  of  the  original, 

c)  A  list  of  all  otner  Images  which  were  derived  from 
this  picture,  and  the  necessary  retrieval  data, 
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VIII. 0  PHILOSOPHICAL  remarks 


S  o  nr  •  general  remarks  about  the  Imag*  processing 
ohllosophy  are  an  appropriate  conclusion  to  this 
dissertation.  Thera  are  many  difficult  problems  In  this 
f  I  »  1  d »  which  can  b*  made  either  harder  or  easier  depending 
on  the  iioproaoh  taKen  to  solve  them,  Whl|e  It  Is  recognized 
that  models  for  the  camera  and  scene  ar®  Important,  one 
cannot  depend  totally  upon  accurate  calibration  of  these 
models  to  eliminate  tna  registration  problems,  whether  the 
working  environment  Is  the  surfaoe  of  Mars,  the  9urfaee  of  a 
f  |at  tab  le«  ora  highway. 

It  Is  Important  that  techniques  be  developed  which  are 
sufficiently  general  that  they  can  be  applied  to 
unanticipated  environments,  It  Is  believed  that  most  of  the 
teohnlaues  In  this  oaoer  can  satisfy  this  requirement  by 
suitably  changing  the  various  models, 

The  unmanned  Space  exploration  programs  should  provide 
excellent  application  areas  for  computer  Image  processing, 
since,  as  the  spacecraft  probe  deeper  Into  space,  the  needs 
for  onboard  Image  Processing  will  Increase, 

A||  exploration  of  the  surfaoe  of  other  planets  w|th(n 
the  foreseeable  future  must  be  done  by  unmanned  landing 
vehicles,  Any  Intelligent  exploration  of  other  planets  will 
require  some  level  of  onboard  Image  orrcesslng  oapablllty  In 
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order  that  the  vehicle  c»n  navigate  and  Interaot  with  Its 
•nv  I  ronirent , 

It  Is  hoped  that  the  results  of  this  and  other  research 
In  the  application  of  computers  will  be  used  by  man  to 
better  understand  the  universe  around  h|m#  and  to  free  man 
frorr  the  trivia  of  dally  life#  rather  than  to  provide  men 
the  tool  to  enslave  others, 
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APPENDIX  A, 

DERIVATION  OF  MATRIX  R 


Th|*  appendix  gives  the  derivation  of  the  matrix  R  of 
eauetlon  (2,4),  given  calibration  data  which  describe  the 
position  and  orientation  of  the  camera, 

Th*  rotation  matrix  R  Is  derived  In  several  stops  from 
the  fleomotrlc  calibration  data,  The  f  |  r  3 1  step  derives  a 
rotation  matrix  R0  whloh  rotates  the  pr|nolpa|  ray  Pp  of  the 
camera,  g|v*n  In  scans  coordinates,  Into  point  (0,0, w)  which 
Is  assumed  to  be  the  center  of  the  Image  plane,  given  In 
camera  coordinates, 

(0,0,  |  I  Pd  I  I >'  »  Rl(Po)  <A,n 

This  rotation  matrix  Is  easily  constructed  by  first  rotating 
Pd  ln  th®  Diane  by  matrix  R1  Into  (x',0,z)',  and  then 

rotating  (x',0,z)'  |n  the  x-z  Plane  by  matrix  R2  Into 
(0,0,  zM*,  The  matrices  R0»  Rl,  and  R2  are  defined! 

x/k  y/k  0 

Rl  =  -y/k  x/k  0  (A, 2) 

0  0  1 
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where  k*sqr t< x*2*y*2  ) , 

i/h  0  -k/h 

R2  s  0  1  0  (A, 3) 

k/h  0  z/h 

where  h  =  sqrt{kt2*z,2)  *  sqrt< xf2+y*2  +  z*2 ) , 

Then  we  define  R0  «  R1»R2,  Rfl  Is  not  unique,  since  a 
rotation  around  the  principal  ray  Is  still  possible,  This 
ambiguity  Is  solved  bv  soeelfylns  the  direction  of  one  other 
vector  In  both  scene  and  Image  coordinates,  Given  vector  V 
In  the  scene  and  the  angle  a  between  the  u-axls  and  the 
projection  of  V  on  the  Image  plane  the  rotation  matrix  R3  Is 
def I ned i 


R0(V)  ■  ( x, y i z  >  i 


(A. 4) 


If  the  d  I  reot  |  on  of  v  In  the  Image  plane  Is  0,  then  eos<1>  s 
x/||x.y||,  and  sln<0>  *  y/l|x,y||,  To  orient  the  vector 

R0(V>  In  the  direction  «»  It  |s  sufficient  to  rotate  RZMV) 
by  an  angle  of  «-(3,  Rotaxicn  matrix  RJ  Is  then  deflnedi 

cost®-!3)  s | n ( «-P )  0 

R3  *  -slnte-f3)  c  o  s  ( o  •  (3 )  0  (A, 5) 

0  0  1 

Finally*  the  matrices  R4  and  R  are  defined! 

R4  =  R3  *  R0  (A, 6) 

R  =  I nv  <  R4 )  (A, 7) 


For  Mariner  6  and  7  olctures  of  Mars*  the  camera  position  Is 


*s 
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derived  from  the  latitude  and  longltuda  of  the 
sub-spacecraft  point  on  tha  aurfaoo,  and  tha  altltuda  of  the 
spacecraft,  The  orientation  of  the  camera  Is  determined 
from  the  latitude  end  longitude  of  point  Pp,  the 
Intersection  of  the  principal  optic  ray  with  the  surface, 
and  the  angle  In  the  Image  plane  between  the  u-axls  and  a 
nor th-Do  I  nt Ing  tangent  vector  at  point  Pp  projected  op  the 
Image  plane,  If  Pp  Is  given  In  spherical  coordinates  asi 


Pp  s  r  *  (  ccs( long)«cos( lat)  (A,6) 

#  s I n  < | ong ) «cos ( I  a t  > 

, s|n( I  at ) ) ' 


where 

1  at 

1  s 

1  at  1 tude 

i  long 

Is  longitude, 

and  r 

1  s  tht 

rad  1  us 

of 

the 

sphere 

(Mars)  i 

then  the  north 

-oo Int 1 ng 

tangent 

veotor 

to 

the 

sphere 

at  this 

point  | s l 

V  »  (  -cos( |ong)*s|n£ |at)  (A, 9) 

, -s | n(  |  ong >*s  |  r»(  | at ) 
i  cos  < I  at  > ) ' 


APPENOIX  B 

THE  effects  of  non.translatiokal 
geometric  distortions  on  CORRELATION 

Th|s  appendix  analyzes  ‘'.he  effects  of  non-trans I  at | ona I 
geoiratrlo  distortions  bstwaan  the  windows  of  two  Images  on 
the  value  of  the  correlation  funotlon,  An  upper  limit  on 
the  size  of  the  correlation  window  Is  derived  as  a  function 
of  the  magnitudes  of  the  higher  order  errors  and  the  spatial 
frequency  Information  In  the  windows,  It  Is  shown  that  the 
offsets  of  non- tr ans I  at ! ona I  geometric  distortion  will  be 
less  If  the  Images  contain  primarily  low  spatial  frequencies 
rather  than  high  soat|a|  frequencies. 

A , 1  ROTATION  AND  SCALE  CHANGE 

Rotation  and  sca|e  factor  errors  between  correlation 
windows  can  be  modeled  with  a  first  order  misalignment 
mode  I  * 

u*  8  a»u  *  b*v  +  c  ■  Su  *  <u*cos(«)  v*sln(*))  tC.la) 

v»  *  d*u  ♦  e*  +  f  s  Sv  *  ( v*cos ( P )  -  u#sln(l3)>  <B.lb) 
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where 


a  s 

arctan<b/a> 

<B.2a> 

P  s 

arcta n( d/e ) 

(B.2b) 

Su 

S  Sqrt<at2+b»2 ) 

<  B  ,  2e ) 

Sv 

s  sart(d»2+et2) 

(B,2d) 

Presumably!  the  rotation  angles  «  and  P  are  small  and  the 
scale  factors  Su  and  Sv  are  approximately  unity, 

The  analysis  of  the  alignment  errors  between  points  In 
two  windows  whose  centers  are  Properly  aligned  (l,e,  csfs2) 


will  be  presented  for  the  special  ease  of: 

S  =  Su  a  Sv  and  orsp 
| | u' -u#  v' -v | 1 1 2 

=  (u«(S*cos<«>-1)  ♦  v*S*s  I n< «>  >  f2  ( B , 3a ) 

♦ ( v* ( S*cos ( « 5 -l )  -  u#S#s I n( « M  »2 
=  I  I  u  #  V | | ♦ 2  •  c  <1-S)»2  +  2*S# ( l*oos { a ) )  ]  (B,3b) 

Hence  we  have  a  linear  model,  l,e, 

I | u' "u* v ' - v | |  s  0(a,S)  •  |  | u , v I |  (  B  ,  4  ) 

$(«,1>  =  2*sin(«/2)  (B,5a) 

Q(0,S)  *  |  1-S  |  (B,5b) 


If  the  centers  of  two  correlation  windows  are  properly 
aligned,  then  the  misalignment  error  due  to  rotation  and 
scale  change  at  a  Point  n  pixels  from  the  center  |s  Q#n, 
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Thus  far  we  have  only  estimated  the  magnitude  of  the 
error  In  the  position  of  cor  respond | ng  pixels  based  on  our 
knowledge  of  the  distortion  process,  Me  now  want  to  know 
the  effect  of  th|s  misalignment  on  the  value  of  the 
correlation  f unc 1 1  on « 

Flsohler  assumes  (flsehler  Cl’71^)  that  adjacent  pixels 
are  statistically  Independent,  so  that  there  Is  no 
contribution  to  the  oroduot  term  E(X*Y>  by  multiplying 
pixels  which  do  not  geometrically  match,  Flsehler  then 
derives  limits  on  the  size  of  the  correlation  window  as  a 
function  of  the  rotation  and  scale  change,  H|s  derivation 
Is  re  1 ntarpreted  as  f  o I  |  o  w  s I 

Assume  that  we  have  two  arrays  of  pixels  XCI,J3  and 
Y[I»J3  which  are  geomstr I ea | I y  |n  alignment,  Suppose  that 
we  have  a  geometric  distortion  of  the  ooordlnate  system  such 
that  a  new  array  of  pixels  i  Is  created,  where  eaoh  pixel 
2[!#J3  Is  derived  by  bilinear  Interpolation  (the  same 
derivation  holds  for  higher  order  Interpolation  and  weighted 
averages  In  general)  from  a  point  at  (u,v)  In  array  Y, 

Let  k s C u 3 ,  |  « C  v D I  then 

2CI,J3  *  a* Y  C  k ,  |  ]  *  b»Y  C  k  +  1 , I  3  ♦  c*YCk, |+13  ♦  d* YC k+1 , I +1 3 ( B  ,  6 ) 
where  a,b,c,  and  d  satisfy  the  conditions  of  equations 


Let  us  assume  for  simplicity  (w|th  no  real  loss  of 
generality)  that  the  Images  have  been  normalized  to  zero 
mean  Intensity  values,  F  | s  c  h  |  e  r ' s  assumption  of 
Independence  than  Implies  that  If  |  u- 1 | >1  or  |v-j|>i  then 
Z[|,J3  was  derived  from  4  points  of  Y  which  are  Independent 
of  YCl.j]  and  *CI|J3.  Hence,  such  points  contribute 

E(X)*E(Y)*0  to  the  sum  of  the  XC I , j 3*Y[ 1 , J 3  In  equation 

(3,5)  , 

If  I u- I | <1  and  |  v-J | <i  then 

sum  X C | , j 3 *2 [  | ,  j ]  -  sum  aC I , J3*X£ I , J 3* YC I , J]  (B,7a) 

I  *  J  I .  J 

*  ( 1-a C I > J3 ) *X[ | > J 3* ( 3  Independent  points  of  Y) 

*  sum  aC 1 ,  J 3*XC I , J 3*YC I »  j  3 ,  (B,7b) 

I  *  J 

and  there  exists  an  A  such  that  0<A<1  and 

sum  X [  1  , J  3*2  C  | ,  J  3  =  sum  ( aC I , J 3  *  X  C  I  ,  j 3*YC I , j 3 )  (B,8a) 

I »  J  I  i  j 

s  A  #  sum  ( XC I »  J  3 • Y [  I  , j  3  )  (B,8b) 

I  *  J 

E(X*Z)  s  A»E ( X* Y ) ,  (P.9) 

Therefore  the  only  effect  of  the  distorted  coordinate 
system  Is  to  attenuate  the  t(X*Y)  term  which  attenuates  the 
value  of  the  correlation  function,  Of  course,  the  value  of 
A  aeoerds  on  the  magnitude  of  the  geometric  distortion 
errors  of  equation  (8,3) , 
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u  »«*H**i*?  »«Mt«!P3R  H-t  r^iCTiaMtt©fl|Qrt?'ii<»^,Ei<BSISfS^^  YFr“ 


A  limit  on  the  size  of  the  correlation  window  Is 
derived  from  eauatlon  (0.4)  by  | I m » 1 1  no  the  maximum  error  to 

one  o I xe I l 


n  a  1  | u#  V  1  | 

(8,10a) 

*  I  1 u'*U»  v' 

-v| |  /  U(«,S) 

(8,10b) 

«  1/Q(«,S) 

(8,10c) 

For  Mariner  6  and  7 

plotures#  the  maximum 

comb  1 ned 

rotation  errors  between  two  Images  are  about  2  degrees  and 
the  eca|e  changes  are  less  than  about  2X,  Therefore# 

nf2  <  l/(,02»2  ♦  2e(l,02)*(l-cos(2) ) )  (8,11) 

or  n  <  25 

Flschler's  Independence  assumption  essentially  says 
that  the  autocorrelation  functions  for  X  and  Y  are  one  at 
u  =  0  and  v c 0  and  zero  elsewhere,  We  Know  that  reasonable 
pictures  do  net  have  such  autocorrelation  surfaces,  In 
particular,  the  autocorrelation  surfaces  have  central  peaks 
whose  width  depends  on  the  low  frequency  power  of  the  power 
spectra  of  the  Images,  Consedusnt | y#  knowing  the 

autocorrelation  function  tells  how  large  the  errors  In  the 
alignment  of  two  Images  can  be  before  the  cross  correlation 
Is  adversely  changed.  If  one  has  a  choice  of  which  areas  In 
the  two  Images  to  a|!gn,  then  the  areas  can  be  chosen  on  the 
basis  of  having  broadly  peaked  autocorrelation  functions, 
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